[PDF] Jellyfish: A Large Language Model for Data Preprocessing | Semantic Scholar (2024)

It is shown that for use cases that do not allow data to be shared with third parties, open-source LLMs can be a viable alternative to hosted LLMs given that a small amount of training data or matching knowledge is required.

This survey aims to address the gap by consolidating recent progress in these areas, offering a thorough survey and taxonomy of the datasets, metrics, and methodologies utilized, while providing some insights for future research directions in this vital and rapidly evolving field.

The feasibility of LLMs as a mechanism for quantitative knowledge retrieval to aid data analysis tasks such as elicitation of prior distributions for Bayesian models and imputation of missing data is explored.

The proposed Unicorn, a unified model for generally supporting common data matching tasks, adopts a mixture-of-experts model that enhances the learned representation into a better representation and can achieve better performance on most tasks and on average, compared with the state-of-the-art specific models trained for ad-hoc tasks and datasets separately.

An LLM-based framework for data preprocessing is proposed, which integrates cutting-edge prompt engineering techniques, coupled with traditional methods like contextualization and feature selection, to improve the performance and efficiency of these models.

The Platypus family achieves strong performance in quantitative LLM metrics across model sizes, topping the global Open LLM leaderboard while using just a fraction of the fine-tuning data and overall compute that are required for other state-of-the-artfine-tuned LLMs.

Prefix-tuning is proposed, a lightweight alternative to fine- Tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which is called the prefix.

Batch prompting, a simple yet effective prompting approach that enables the LLM to run inference in batches, instead of one sample at a time, is proposed, which reduces both token and time costs while retaining downstream performance.

A general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation, and finds that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

This work proposes and evaluates Evaporate, a prototype system powered by large language models, and proposes an extended implementation, Evaporate-Code+, which achieves better quality than direct extraction, and proposes an extended implementation, Evaporate-Code+, which achieves better quality than direct extraction.

A novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models is proposed and, by leveraging this new KD method, the plenty of knowledge encoded in a large “teacher” BERT can be effectively transferred to a small “student” TinyBERT.

Rotom is a multi-purpose data augmentation framework for a range of data management and mining tasks including entity matching, data cleaning, and text classification that automatically learns a policy for combining examples from different DA operators, whereby combinatorially reduces the hyper-parameters space.

Low-Rank Adaptation, or LoRA, is proposed, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.

It is found task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings actually yields stronger performance in all settings.

...

...