Earlier this week, Meta announced the release of <a href="https://ai.meta.com/llama/">Llama 2</a>. Llama 2 includes both a base pre-trained model and a fine-tuned model for chat available in three sizes. While the performance of the pre-trained model is impressive, fine-tuning the base Llama-2 model can unlock even greater performance on most language tasks. This process can be quite complicated, from reserving the right compute resources to properly implementing the training and inference code.
Scale's new open-source <a href="https://github.com/scaleapi/llm-engine">LLM Engine</a> repository makes this process easy and accessible. Let's dive into how you can use this API and explore some examples.
<pre class="language-c"><code>fix

fix</code></pre>
<pre class="language-python"><code>from llmengine import FineTune

response = FineTune.create(
 model="llama-2-7b",
 training_file="s3://my-bucket/path/to/training-file.csv",
)

print(response.json())</code></pre>
You can also follow the code examples in this <a href="https://github.com/scaleapi/llm-engine/blob/main/examples/finetune_llama_2_on_science_qa.ipynb">notebook</a>.
<h1>Dataset</h1>
For this example, we will use <a href="https://scienceqa.github.io/">ScienceQA</a>, a popular dataset consisting of a diverse set of multiple-choice science questions. Each question may have textual context and image context and contains a thorough explanation and lecture supporting the solution.
<img src="https://lh3.googleusercontent.com/hB2u6hfFZMXlmtwOLgT3-wrJKdlyAdd9MssPkfvoysG2KRdnafhfxHfJ4IVRKDt1Bj71LikV-vtmXBK1P11sOqS8ea7nGnD1_1mVIKAIcQn4OIbKNKESnpSobwRFBsnTAbYT_7t_RARN83xNiwGRitk" width="624" height="260">
An example sample from ScienceQA
Currently, LLM Engine supports fine-tuning on prompt-completion pairs. Let's first convert this dataset into the supported format, a CSV with two columns: <code>prompt</code> and <code>response</code>.
Before you get started, install the required dependencies.
<code>pip install datasets==2.13.1 smart_open[s3]==5.2.1 pandas==1.4.4</code>
We can load the dataset from Hugging Face, and observe the dataset's features.
<pre class="language-python"><code>from datasets import load_dataset
from smart_open import smart_open
import pandas as pd
 
dataset = load_dataset('derek-thomas/ScienceQA')
dataset['train'].features</code></pre>
A commonly used format for feeding ScienceQA examples is:
<pre class="language-markup"><code>Context: A baby wants to know what is inside of a cabinet. Her hand applies a force to the door, and the door opens.
Question: Which type of force from the baby's hand opens the cabinet door?
Options: (A) pull (B) push
Answer: A.</code></pre>
Since the format of <code>options</code> in the Huggingface dataset is only a list of possible answers, we need to convert this list into the example format from above by adding the enumeration prefix.
<pre class="language-python"><code>choice_prefixes = [chr(ord('A') + i) for i in range(26)] # A-Z
def format_options(options, choice_prefixes):
 return ' '.join([f'({c}) {o}' for c, o in zip(choice_prefixes, options)])</code></pre>
&nbsp;Now, let's write the formatting function to turn a single sample from this dataset into a <code>prompt</code>&nbsp;and <code>response</code> to feed into the model.
<pre class="language-python"><code> def format_prompt(r, choice_prefixes):
 options = format_options(r['choices'], choice_prefixes)
 return f'''Context: {r["hint"]}\nQuestion: {r["question"]}\nOptions:{options}\nAnswer:'''
 
def format_response(r, choice_prefixes):
 return choice_prefixes[r['answer']]</code></pre>
Finally, let's construct the dataset. Note that some samples in ScienceQA only have an image for context - we'll be skipping those in this example as Llama-2 is purely a language model, and cannot accept image inputs.
<pre class="language-python"><code> def convert_dataset(ds):
 prompts = [format_prompt(i, choice_prefixes) for i in ds if i['hint'] != '']
 labels = [format_response(i, choice_prefixes) for i in ds if i['hint'] != '']
 df = pd.DataFrame.from_dict({'prompt': prompts, 'response': labels})
 return df</code></pre>
LLM Engine supports training with a training and validation dataset. If you only provide a training dataset, the LLM Engine will randomly split 10% from the dataset for validation. Splitting your dataset prevents the model from overfitting on your training data, resulting in poor generalization to live data it might see during inference.
These dataset files must be stored in a publicly accessible URL so that LLM Engine can read them. For this example, we will be saving the datasets to s3. We also have the preprocessed <a href="https://gist.githubusercontent.com/jihan-yin/43f19a86d35bf22fa3551d2806e478ec/raw/91416c09f09d3fca974f81d1f766dd4cadb29789/scienceqa_train.csv">train dataset</a> and <a href="https://gist.githubusercontent.com/jihan-yin/43f19a86d35bf22fa3551d2806e478ec/raw/91416c09f09d3fca974f81d1f766dd4cadb29789/scienceqa_val.csv">validation dataset</a> publicly&nbsp;available in Github Gists - you can directly replace <code>train_url</code> and <code>val_url</code> with these links.
<pre class="language-python"><code>train_url = 's3://...'
val_url = 's3://...'
df_train = convert_dataset(dataset['train'])
with smart_open(train_url, 'wb') as f:
 df_train.to_csv(f)
 
df_val = convert_dataset(dataset['validation'])
with smart_open(val_url, 'wb') as f:
 df_val.to_csv(f)</code></pre>
Now, we can start fine-tuning via the LLM Engine API.
<h1 dir="ltr">Fine-Tuning</h1>
First, we need to install LLM Engine.
<code>pip install scale-llm-engine</code>
Next, you'll need to set up your Scale API key. Follow the instructions in the&nbsp;<a href="https://github.com/scaleapi/llm-engine#-quick-start">README</a> to acquire your unique API key. Advanced users can also follow the guide for <a href="https://scaleapi.github.io/llm-engine/guides/self_hosting/">self-hosting LLM Engine</a> to avoid needing a Scale API key.
<pre class="language-python"><code>import os
os.environ['SCALE_API_KEY'] = 'xxx'</code></pre>
Once you have everything set up, fine-tuning your model requires only one API call. Today, we'll be working with the 7 billion parameter version of Llama-2, which proves powerful enough for most use cases.
<pre class="language-python"><code>from llmengine import FineTune
response = FineTune.create(
 model="llama-2-7b",
 training_file=train_url,
 validation_file=val_url,
 hyperparameters={
 'lr':2e-4,
 },
 suffix='science-qa-llama'
)
run_id = response.fine_tune_id</code></pre>
With the <code>run_id</code>, you can monitor the status of your job and get live updates on per-epoch metrics, like train and validation loss.
ScienceQA is a large dataset, so it may take an hour or two for training to complete.
<pre class="language-python"><code>while True:
 job_status = FineTune.get(run_id).status
 # Returns one of `PENDING`, `STARTED`, `SUCCESS`, `RUNNING`,
 # `FAILURE`, `CANCELLED`, `UNDEFINED` or `TIMEOUT`
 print(job_status)
 if job_status == 'SUCCESS':
 break
 time.sleep(60)

#Logs for completed or running jobs can be fetched with
logs = FineTune.get_events(run_id)</code></pre>
<h1 dir="ltr">Inference &amp; Evaluation</h1>
Once you are done fine-tuning, you can start generating responses to any input. However, before that, let's ensure the model exists and is ready to accept inputs.
<code>ft_model = FineTune.get(run_id).fine_tuned_model</code>
It may take a few minutes for your first inference call to be served. After that, inference should be relatively quick.
Now, let's evaluate the performance of our Llama-2 model fine-tuned on ScienceQA.
<pre class="language-c"><code>fix

fix</code></pre>
<pre class="language-python"><code>import pandas as pd
 
#Helper a function to get outputs for fine-tuned model with retries
def get_output(prompt: str, num_retry: int = 5):
 for _ in range(num_retry):
 try:
 response = Completion.create(
 model=ft_model, 
 prompt=prompt, 
 max_new_tokens=1, 
 temperature=0.01
 )
 return response.output.text.strip()
 except Exception as e:
 print(e)
 return ""
 
#Read the test data
test = pd.read_csv(val_url)
 
test["prediction"] = test["prompt"].apply(get_output)
print(f"Accuracy: {(test['response'] == test['prediction']).mean() * 100:.2f}%")</code></pre>
This achieves 82.15% accuracy. This is pretty good! Let's see how this compares with the original base Llama-2 model. Since the pre-trained model was not fine-tuned on these examples, we need to provide an example in the prompt so the model learns to adhere to the format we expect from the responses. We also see how things compare against fine-tuning a model of similar size, Mosaic's&nbsp;<a href="https://www.mosaicml.com/blog/mpt-7b">MPT</a>.
<table style="border-collapse: collapse; width: 100%; height: 58.7814px;" border="1"><colgroup><col style="width: 25%;"><col style="width: 25%;"><col style="width: 25%;"><col style="width: 25%;"></colgroup>
<tbody>
<tr style="height: 19.5938px;">
<td style="height: 19.5938px;">ScienceQA Validation</td>
<td style="height: 19.5938px;">MPT-7B</td>
<td style="height: 19.5938px;">Llama 2-7B</td>
<td style="height: 19.5938px;">GPT-3.5*</td>
</tr>
<tr style="height: 19.5938px;">
<td style="height: 19.5938px;">Pre-trained (1-shot)</td>
<td style="height: 19.5938px;">44.87%</td>
<td style="height: 19.5938px;">55.56%</td>
<td style="height: 19.5938px;">74.44%</td>
</tr>
<tr style="height: 19.5938px;">
<td style="height: 19.5938px;">Fine-tuned</td>
<td style="height: 19.5938px;">81.43%</td>
<td style="height: 19.5938px;">82.15%</td>
<td style="height: 19.5938px;">N/A</td>
</tr>
</tbody>
</table>
*Results from <a href="https://arxiv.org/abs/2302.00923">MM-COT paper.</a>
As we can see, fine-tuning makes a huge difference in the case of both language models. The performance gain from fine-tuning Llama-2 on ScienceQA was a&nbsp;26.59%&nbsp;absolute difference! This is in addition to the fact that inference with a fine-tuned model is cheaper than few-shot example prompts, due to shorter prompt length. This fine-tuned Llama-2-7B model also outperforms GPT-3.5, a 175 billion parameter model!
We can also see that the new Llama-2 model outperforms MPT in both the fine-tuned and few-shot prompting settings, showcasing its strength as both a base and fine-tunable model.
As a side experiment, we also used LLM Engine to fine-tune and evaluate LLAMA-2's performance on several tasks from <a href="https://gluebenchmark.com/">GLUE</a>, a set of commonly used NLP benchmark datasets.
<table style="border-collapse: collapse; width: 100%;" border="1"><colgroup><col style="width: 20.0284%;"><col style="width: 20.0284%;"><col style="width: 20.0284%;"><col style="width: 20.0284%;"><col style="width: 20.0284%;"></colgroup>
<tbody>
<tr>
<td>Model</td>
<td>MRPC</td>
<td>RTE</td>
<td>COLA</td>
<td>WNLI</td>
</tr>
<tr>
<td>Llama 2 (Pre-trained)</td>
<td>68.62%</td>
<td>73.65%</td>
<td>71.33%</td>
<td>63.38%</td>
</tr>
<tr>
<td>Llama 2 (Fine-tuned)</td>
<td>89.21%</td>
<td>89.89%</td>
<td>86.76%</td>
<td>71.83%</td>
</tr>
<tr>
<td>MPT (Fine-tuned)</td>
<td>87.75%</td>
<td>87.36%</td>
<td>86.48%</td>
<td>70.42%</td>
</tr>
</tbody>
</table>
<h1 dir="ltr">Conclusion</h1>
Fine-tuning makes a huge difference in unlocking greater performance from base pre-trained models. Now you are all set to unleash the true potential of your fine-tuned model and witness the magic of powerful AI-generated responses. Happy fine-tuning! 🚀

Deploy and fine-tune models using our open-sourced LLM Engine Repo. Learn how in our latest blog. 

Fine-Tune Llama 2 with Scale's LLM Engine Repository

How to Fine-Tune Llama 2 With LLM Engine

Today, every company is investing in AI initiatives after the explosion of generative AI in the past six months. Those that don&rsquo;t will get left behind. Yet many companies don&rsquo;t know how to implement AI into their business operations. Customizing models is extremely difficult, requiring expertise in ML Infrastructure and experience with a large number of software packages. Often these software packages are cobbled together and don&rsquo;t achieve the desired results. Enterprises rarely have the internal expertise or time to ensure AI is done right.&nbsp;
So today, we&rsquo;re launching an <a href="https://scale.com/custom-llms">open-source platform</a> for customizing large language models (LLMs) for enterprise use cases. The platform allows enterprises to customize open-source LLMs to achieve optimal performance for their specific use cases using their own data. Businesses can more easily build applications for their highest-priority generative AI use cases. Fine-tuning open-source models with Scale will give organizations better performance for their specific use cases, increased customizability, and enhanced safety at a lower total cost than other solutions.
As part of our efforts, Scale is proud to be a <a href="https://ai.meta.com/llama/#partnerships" target="_blank" rel="noopener">Meta Llama 2 Launch Partner</a>, and we're excited to make it easier to customize open-source LLMs like Llama 2. Scale CEO and founder Alexandr Wang also signed a <a href="https://about.fb.com/news/2023/07/llama-2-statement-of-support/" target="_blank" rel="noopener">statement in support</a>&nbsp;of Meta&rsquo;s open approach to AI that enables everyone to benefit from the technology and brings additional levels of visibility, scrutiny, and trust.&nbsp;
<h2 dir="ltr">What Scale is Making Possible</h2>
With this new offering, enterprises can customize open-source models with just a few lines of code leveraging their own data. For more demanding model customization needs, Scale&rsquo;s fine-tuning platform, combined with our <a href="https://scale.com/data-engine">Data Engine</a>, will optimize model performance for the most challenging business use cases. The Scale Data Engine powers the most advanced LLMs and generative models through world-class RLHF, data generation, model evaluation, safety, and alignment, making it easy for every enterprise to adopt industry-leading generative AI.
Scale helps insurance companies summarize and classify claims more efficiently, retail and e-commerce companies improve the shopping experience through personalized recommendations, and wealth managers increase returns by quickly summarizing and identifying the most relevant trends from financial data. Achieving this performance at these specific tasks is not possible without highly capable base models that are fine-tuned with high-quality data.
Our new platform enables:
<ul>
<li dir="ltr" aria-level="1">
Performance: Models customized with proprietary data perform better at high-priority tasks.
</li>
<li dir="ltr" aria-level="1">
Customizability: Customize models to specific use cases, and shape how they respond to common scenarios so they can be an effective extension of the brand.
</li>
<li dir="ltr" aria-level="1">
Safety: Enhance the safety of models with human-in-the-loop testing, evaluation, and monitoring.
</li>
</ul>
<h2>How to Get Started</h2>
To support an open innovation approach to AI, Scale is providing an <a href="https://github.com/scaleapi/llm-engine">open-source package</a> and open APIs for model fine-tuning and hosting, along with clear documentation on how to use these APIs. The package and APIs empower any organization, no matter their expertise, to get up and running quickly and easily. Enterprises that need an additional performance boost for their specific use cases can leverage the power of human experts to enhance their data with the Scale Data Engine. Ultimately, this platform will deliver fine-tuned open-source models for organizations&rsquo; most important use cases and for specific domains to encourage the adoption of generative AI across industries.
Get started by downloading the&nbsp;<a href="https://github.com/scaleapi/llm-engine">open-source repo from Github,</a> or if you are looking to optimize your models with our enterprise model customization platform and the power of the Scale Data Engine, <a href="https://scale.com/custom-llms">contact us</a> today.

Scale is proud to be a Meta Llama 2 launch partner and we're excited to make it easier to customize open-source LLMs like Llama 2 with our platform for customizing open-source large language models (LLMs) for enterprise use cases.

Scale Unlocks Open-Source LLMs With New Platform and Partnership with Meta

<a href="https://scale.com/">Scale AI</a>, the data infrastructure for AI, has been selected by the U.S. Army and the <a href="https://www.diu.mil/">Defense Innovation Unit</a> (DIU) to deploy their Data Engine in support of the Army&rsquo;s Robotic Combat Vehicle (RCV) program. The effort is a joint collaboration between DIU and the Army&rsquo;s Program Executive Office Ground Combat Systems (PEO GCS) leveraging innovative Other Transaction Authority for the novel application of Scale&rsquo;s commercial technology to the RCV program.
The Army has determined that the next stage of navigating high-risk environments requires advanced technology, like trusted autonomous systems, to safely conduct reconnaissance and related tasks. Scale will play a vital role in the development of these systems by providing its Data Engine platform to support raw data collection, curation, and labeling, as well as AI model development and test and evaluation for autonomous navigation software and computer vision algorithms.&nbsp;&nbsp;&nbsp;
Mitigating risk for combat troops is paramount in military operations, and Scale is committed to supporting the U.S. Army&rsquo;s efforts to ensure the safety of our service men and women. With a strong background in the autonomy space, Scale has established partnerships with notable customers such as General Motors, Lyft, and Toyota. Leveraging their deep expertise, Scale will provide robust support for the RCV program, delivering proven commercial technology tailored for defense applications, including off-road terrain, extreme weather conditions, and contested or degraded communications.
Whether you&rsquo;re a fighter pilot in the Air Force, or an Armor Officer in the Army, nothing replaces real world training events when developing combat-ready Airmen or Soldiers; synthetics and simulation can only get you so far and AI for autonomy is no different. Scale is excited to partner with the U.S. Army who will benefit from our deep expertise in autonomy combined with our <a href="https://scale.com/data-engine">Data Engine </a>technology, to deliver the data management plans and platforms DoD autonomy programs need.&nbsp;
Come talk to us at the GEOINT conference in St. Louis, MO from 22-24 May. We will be at booth #633 or check us out on LinkedIn!
Learn more here: <a href="https://scale.com/data-engine">https://scale.com/data-engine</a>
&nbsp;

Scale AI, the data infrastructure for AI, has been selected by the U.S. Army and the Defense Innovation Unit (DIU) to deploy their Data Engine in support of the Army’s Robotic Combat Vehicle (RCV) program. 

Scale

Scale AI chosen by the U.S. Army for Robotic Combat Vehicle Program

<h2 dir="ltr">Background</h2>
Object tracking in aerial imagery is largely an unsolved problem in the AI overmatch future powered by Generative AI and platforms like <a href="https://scale.com/donovan" target="_blank" rel="noopener">Scale Donovan</a>; expert algorithms ATR and chain-of-custody play a crucial role as tools enabling the models to expand out from a pure text and structured data domain.
Although there exists extensive research on object tracking in the academic realm, those solutions are largely targeted at accurate assignment of track ID&rsquo;s on top of fixed object detector outputs. These outputs come from a nearly perfect detector on common commercial footage. This footage typically comes from a camera that is mounted front-facing on the ground in a fixed position with fixed zoom levels.&nbsp;
However, the needs of many Federal customers do not align with these cookie-cutter solutions. They often require aerial imagery and full motion video capabilities, which present additional challenges, on top of accurate assignment of track ID&rsquo;s as in the commercial realm:&nbsp;
<ol>
<li dir="ltr" aria-level="1">
The base object detectors are less performant, meaning there is significant room for the tracker to use temporal information to reduce false positives and false negatives.
</li>
<li dir="ltr" aria-level="1">
The camera moves quite a bit in terms of position (i.e., panning) and zoom levels.
</li>
<li dir="ltr" aria-level="1">
Long-term re-identification of targets is non-trivial and mission-critical.&nbsp;
</li>
</ol>
<h2 dir="ltr">Scale&rsquo;s Solution: A Suite of Tracker Post-Processors</h2>
To address these problems, Scale offers a suite of tracker post-processors, each of which can be turned on or off.&nbsp; We optionally package these on top of our own custom, base tracker which achieves state-of-the-art balance between precision and recall when tracking in the presence of camera movement.&nbsp;
These post-processors can be used alone to enhance an existing tracker, or used as an end-to-end tracking package on top of an existing detector. This package can be called in one line of code per frame after each object detector inference (for users using the package) or base tracker inference (for users using just the post-processors).&nbsp;
<h2 dir="ltr">Sample Demo MP4 Results</h2>
Below, we compare our Scale Tracker packaged with our custom base tracker (25FPS on the left) with two of the best publicly available trackers - one that is faster (BYTETrack at 30FPS in the middle) which has comparatively poor track ID permanence, and one that is slower with slightly better track ID permanence than BYTETrack (SmileTrack at 10FPS on the right), although still not as good as Scale Tracker&rsquo;s track ID performance:
<img title="image1.gif" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/01b73d81-ae6e-4a6d-b744-8cdcc98aedce" alt="" width="1200" height="300">
<img title="image2.gif" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/3945304f-e9e7-4aaf-a166-ff9f2ab89934" alt="" width="1200" height="300">
<img title="image3.gif" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/8dd6967b-f7bb-414d-aa5d-58952facd8d8" alt="" width="1200" height="300">
The Scale tracker:
<ol>
<li dir="ltr" aria-level="1">
Flickers less -- this creates more visually persistent tracks thereby creating a better user experience
</li>
<li dir="ltr" aria-level="1">
Is able to maintain track ID&rsquo;s through periods of erroneous track dropping -- this creates tracks that exhibit more permanence
</li>
</ol>
<h2 dir="ltr">Empirical Results</h2>
<h3 dir="ltr">Academic results</h3>
<img title="image4.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/0691ca39-f375-405a-8081-c8cfaf0484ad" alt="" width="615" height="438">
Note: The metrics for the other methods in the above chart were taken directly from the <a href="https://arxiv.org/pdf/2202.13514.pdf">StrongSORT paper</a>
Even though our tracker is aimed at Federal problems, it actually improves upon the performance of trackers in the commercial realm and achieves state-of-the-art trade-off between speed and performance when paired with BYTETrack, as evaluated by IDF1 (a mixture between precision and recall) on common commercial tracking dataset MOT17.&nbsp;
The chart above demonstrates that ScaleTrack (packaged as our post-processors paired with BYTETrack) achieves a better speed/performance tradeoff than all other methods. BYTETrack and OC-sort are slightly faster, but significantly less performant than ours.&nbsp;
Why the emphasis on the comparison to BYTETrack (and not StrongSORT or SmileTrack)? Because any tracker that cannot be run at near-real time is nearly useless for Full Motion Video (FMV) workflows because most FMV users want algorithms that can be run as they review mission videos. And most cameras collect around 20FPS, meaning only the algorithms to the right of the red line can be of high value to FMV workflows.
<h3 dir="ltr">Federal results</h3>
The above graph is from evaluating an academic dataset. On mission-specific datasets, we outperform even more. As visual evidence of our outperformance over state-of-the-art in Federal datasets, please see the above MP4 sample results.&nbsp;
<h2 dir="ltr">How Did We Achieve These Results? (Technical Deep Dive)</h2>
<h3 dir="ltr">Reduction of flickering</h3>
Detectors basically never have 100% recall in the Federal domain. They often miss objects. Most trackers halt operations when this happens and kill the track. Certain trackers keep the object around in a state that is half dead/half alive for about 5 frames, and provide them the option of being revived if the object appears again.&nbsp;
We went with a different approach. We become skeptical of the detector when a track isn&rsquo;t propagated to the next frame and we try to persist the track by querying the image directly through a two-step process. This two-step process performs bounding box location prediction functions for a given track. The first prediction function is based on kalman information.&nbsp;
However, kalman filters fail sometimes. As a result, Scale did a thorough, detailed study of these failure cases and created a second, custom prediction function in bounding box search space. Given these bounding box hypotheses (from the kalman filter prediction function and our custom prediction function), we use a highly specialized visual feature function (which is not customer-locked or security-locked and therefore can be used by any customer at any security level) to select the bounding box which is most visually consistent with that track&rsquo;s visual history. Based on the level of visual consistency, we either persist the track to the next frame at that selected bounding box&rsquo;s location, or enter it into a state that is not yet dead but not entirely alive (i.e., a bank of tracks from which ReID can draw).
<h3 dir="ltr">Ability to maintain track ID's through short periods of erroneous track dropping</h3>
Sometimes, the flickering reduction mechanisms fail. But it&rsquo;s important that when the track shows up again, it maintains the same ID. So, for a set amount of frames after a track is last seen, we use its kalman filter to project the track over every frame. If there ever exists a new track that is visually consistent with the dead track and that new track exists in a location that is highly likely under the dead track&rsquo;s motion prediction distribution, we assign the dead track&rsquo;s ID to the new track. Thereby we would &ldquo;revive&rdquo; the old track in the new track&rsquo;s location, which allows us to maintain the track ID even though the object was missing for a short while. We refer to this as &ldquo;Local ReID.&rdquo; Outside of handling algorithmic false negatives, this also addresses short-term occlusions (e.g. a vehicle driving underneath an occluding tree).
<h3 dir="ltr">Ability to maintain track ID's through long periods in which the track wasn't present</h3>
This addresses the following use cases:
<ol>
<li dir="ltr" aria-level="1">
Objects coming in and out of the frame, potentially at an entry location significantly different from its exit location
</li>
<li dir="ltr" aria-level="1">
Objects that are occluded for long periods of time (e.g., a person coming in and out of a building)
</li>
<li dir="ltr" aria-level="1">
Camera zoom ins, zoom outs, and pans
</li>
</ol>
In these cases, traditional motion prediction breaks down, meaning traditional motion information prediction algorithms cannot be used to link new tracks with dead tracks. However, the objects rarely significantly change their looks within the span of a video. So, we introduce &ldquo;Global ReID&rdquo; which attempts to link dead tracks to new tracks strictly through a harsh visual filter.&nbsp;
<h3 dir="ltr">Ability to maintain track ID's through medium-length periods in which the track wasn't present</h3>
We further address use-cases 1 and 2 in the above section using a ReID function that is based on custom motion functions mixed with the above mentioned visual filter with time between dead and new tracks as a salient input to the mixture function. We do so in such a way that the larger the gap between these tracks, the less the linking function relies on spatial information and the more it relies on visual information, as there is increased uncertainty in motion as the time gap becomes larger.
&nbsp;

Assign accurate track ID’s and improve performance on top of aerial object detector outputs with video inputs

Scale's Solution for Aerial Object Tracking | Scale AI

Scale's Solution for Aerial Object Tracking

I am excited to join Scale as the General Manager, Public Sector. Over the past 25 years, I have dedicated myself to advancing public and private sector partnerships and leading transformational efforts that bridge the gap between the leading technologists and our federal users who need it. I am thrilled to embark on this new chapter with a company that shares my passion for leveraging cutting-edge technology to help the public sector adapt to the rapidly evolving technological landscape. As a former federal officer who served in far-flung areas of the globe, the work that Scale is accomplishing now is something we had dreamed of a decade ago.&nbsp;We are now living in that future.&nbsp;
Having served with the U.S. Army and Intelligence Community, I have witnessed firsthand the data challenges faced by the United States&ndash;from the tactical to the strategic. They all had one thing in common: our analysts, operators, and decision-makers could be better served by better data. Even two decades ago, my colleagues warned that they would have a problem with understanding, processing, and using the data generated by our forces at scale. And that is the challenge that Scale is solving today. Personally, Scale&rsquo;s commitment to supporting the U.S. government resonates with me, as I come from a family with four generations of public service. I am eager to collaborate with the U.S. government in delivering mission-critical, essential technology from the most innovative technologists.&nbsp;
Scale first caught my attention as a leader in the AI field when I heard about its own <a href="https://scale.com/blog/ukraine-detection" target="_blank" rel="noopener">battle damage</a> analysis of Ukraine; I was so impressed with the scope and the speed of work that was being accomplished on the front lines. Scale&rsquo;s computer vision machine learning is now also complemented by its large language model offering&nbsp;<a href="https://scale.com/donovan">Donovan</a>, Scale&rsquo;s federal-focused AI-power decision-making platform. I have always been a staunch advocate for transformation and innovation grounded in feedback from the end users. I am determined to drive forward efforts at a company that delivers this technology at scale&ndash;pun intended. The immense potential&ndash;and reality&ndash;of what Scale is doing today truly inspires me.
In particular, Scale&rsquo;s credo &ldquo;why not faster&rdquo; deeply resonates with me because the need to adapt and maintain the United States&rsquo; strategic advantage on the global stage has never been more urgent. Additionally, Scale&rsquo;s impressive capabilities&ndash;and its ability to deliver them securely&ndash;is bolstered by its incredible talent. I take great pride in the fact that Scale&rsquo;s employees, leaders and investors value service and support for the public sector.
 Join me at <a href="https://scale.com/careers">https://scale.com/careers</a>.

John joined Scale as the General Manager, Public Sector to accelerate the development of AI applications 

Why John Brennan joined Scale as General Manager, Public Sector

Why I Joined Scale

After the end of WWII and the invention of the atomic bomb during the Manhattan Project, J. Robert Oppenheimer gave an important speech reflecting on the technology he had just built, and in particular, the quantum and irrevocable impact it had on the nature of war:
&ldquo;It is clear to me that wars have changed.
It is clear to me that if these first bombs&mdash;the bomb that was dropped on Nagasaki&mdash;that if these can destroy ten square miles, then that is really quite something.
It is clear to me that they are going to be very cheap if anyone wants to make them.
It is clear to me that this is a situation where a quantitative change, and a change in which the advantage of aggression compared to defense&mdash;of attack compared to defense&mdash;is shifted, where this quantitative change has all the character of a change in quality, of a change in the nature of the world.&rdquo;
These words have great relevance today with AI. We are in the midst of the next great sea change in the nature of war, and thus, the nature of the world.
We live in uncertain times. Geopolitical risk is at&nbsp;<a href="https://www.matteoiacoviello.com/gpr.htm">a 20-year high</a> since the Ukraine war. The United States is facing <a href="https://www.jcs.mil/Media/News/News-Display/Article/613868/dempsey-us-forces-must-adapt-to-deal-with-near-peer-competitors/">near-peer competitors</a> in China and Russia for the first time since the end of the Soviet Union. There is a global battle for ideologies between Western democracy and authoritarian regimes.
All at once, the advent of AI has upended the world. It&rsquo;s a paradigm shift that has granted new superpowers and anything seems possible&mdash;with potentially greater implications on the future of our world than even the internet.
As someone who has been building in the AI ecosystem for more than seven years now, it&rsquo;s exciting to see this technology finally reach its watershed moment. But at the same time, this technology will exacerbate the uncertainty of our times. AI has the potential to unseat incumbents, catapult new leaders, or solidify existing moats. Navigating this new landscape is an existential matter&mdash;for many organizations, for our nation, and for democracy.
The United States, and the democratic world broadly, is confronted with two very real challenges.
First, as I&rsquo;ve <a href="https://alexw.substack.com/p/war?utm_source=profile&amp;utm_medium=reader2">discussed</a> previously, I believe the next era of war and deterrence will be defined by AI. We must ensure the strength of democracy in our AI war against authoritarianism by building the most effective military implementations of AI.
Second, a show of force is necessary, but not sufficient. At the end of the day, the strength of democracy has, and always will, stem from a stronger and faster-growing economy. That&rsquo;s why it is critical that we also bolster our global industries and enterprises with best-in-class capabilities.
The new Scale Donovan and Scale EGP platforms are our answer to these critical challenges&mdash;to empower the U.S. government and enterprises, respectively.
<h2 dir="ltr">Donovan &mdash; Ammunition in the AI War</h2>
AI will define the next era of war and deterrence, just as the atomic bomb did for the last era. The only solution to the AI War is to accelerate our own military implementations and rapidly integrate AI into warfighting. That's why we've funded our own R&amp;D to build Donovan&mdash;and achieve AI overmatch.
<a href="https://scale.com/donovan">Scale Donovan</a> is an AI-powered decision-making platform to help operators understand, plan, and act in minutes instead of weeks. Donovan ingests and understands vast amounts of structured and unstructured data to enable operators and analysts to make sense of any aspect of the real world in minutes using simple natural language. Donovan combines these insights with Scale&rsquo;s proprietary and state-of-the-art models to generate summary reports, COAs, and more to force-multiply staff bandwidth and achieve mission success.
As a longtime leader in AI, I am proud to be delivering this world-changing capability to the United States government with Donovan - the first LLM deployed on a classified network for the XVIII Airborne Corps - ingesting over 100,000 pages of live data to enable actionable insights across the battlefield. The sheer volume of inputs from various formats is a key challenge facing analysts and operators across the federal space. We cannot let the proliferation of information and manual processing be our Achilles heel.
<div class="wistia_responsive_padding" style="padding: 56.25% 0 0 0; position: relative;">
<div class="wistia_responsive_wrapper" style="height: 100%; left: 0; position: absolute; top: 0; width: 100%;"><iframe class="wistia_embed" title="Scale - Donovan Video" src="https://fast.wistia.net/embed/iframe/qugojsb52u?videoFoam=true" name="wistia_embed" width="" height="" frameborder="0" scrolling="no" allow="autoplay; fullscreen"></iframe></div>
</div>
<h2 dir="ltr">EGP &mdash; Unlocking Generative AI for Enterprises</h2>
<a href="https://scale.com/generative-ai-platform">Scale Enterprise Generative AI Platform (EGP)</a> is a full-stack solution for any company looking to deploy Generative AI applications in production. We are helping our customers - like Koch Enterprises - move rapidly from just experimenting with this technology to actually capturing business value from it, through a secure, enterprise-grade platform that is model-agnostic and powered by the industry's leading&nbsp;<a href="https://scale.com/data-engine">data engine</a>.&nbsp;
While Generative AI has captured the imagination of consumers, there are substantial safety, security, and performance challenges when it comes to business use cases. EGP allows customers to develop and deploy Generative AI applications within Scale&rsquo;s infrastructure or in their own virtual private cloud (VPC), so proprietary data never has to leave customers&rsquo; environments. Scale is the only provider delivering Generative AI applications in their own environment, coupled with industry-leading capabilities like Reinforcement Learning with Human Feedback (RLHF) and red-teaming to ensure that model outputs are aligned with customers&rsquo; policies and brand.
Further, EGP powers AI applications across industries. Think copilots for financial advisors to draw upon a much broader range of information when they are educating their clients, or tools for creatives to generate infinitely customizable images and copy, or a chat advisor to guide customers seeking information about retail products. These applications are only the tip of the iceberg - with EGP, companies are empowered to build and develop their own highly specific use cases. By bringing together Generative AI models with their own proprietary data, businesses can unlock new channels for growth and provide instant insights for better, faster decision-making.&nbsp;
<h2 dir="ltr">Looking Ahead</h2>
Right now, we&rsquo;re seeing a lot of AI tourists pretending to be AI natives, taking companies for a ride while they&rsquo;re at their most vulnerable. They&rsquo;re not selling solutions, they&rsquo;re selling vaporware. Everywhere you look, you&rsquo;ll see non-AI companies rebranding and selling technology that they only began to study yesterday. They barely have the knowledge internally to build the products they&rsquo;re selling, and they sure as hell don&rsquo;t have the expertise required to help customers build and implement a long-term, defensible AI strategy.
It&rsquo;s a problem that I care deeply about because the last thing the AI industry needs is more hype. And companies that are claiming AI expertise who haven&rsquo;t spent years in the trenches have the potential to do more harm than good.&nbsp;
The reality is that it takes a lot of hard work to go from experiment to enterprise. Foundation models aren&rsquo;t like a typical consumer software product that works out-of-the-box. From explainability and auditability to PII and proprietary data, there are several critical challenges businesses must address to make AI a reality. There is no established roadmap to follow, which is why only <a href="https://scale.com/ai-readiness-report">21% of enterprises</a> actually have AI models in production. In the federal government, only <a href="https://saic.com/sites/default/files/2023-01/SAIC-Cloud-WP-2023.pdf">14% of executives</a> believe they are &ldquo;very&rdquo; ready for AI.&nbsp;
It's clear that governments and businesses need a partner that provides battle-tested tools and expertise to harness Generative AI in a way that&rsquo;s trusted, secure and moves at the speed of the world around us today. Our new Scale EGP and Scale Donovan platforms were built to solve this problem.
The future of our world hangs in the balance of the current moment. AI has the potential to strengthen global democracies and democratic economies&mdash;or be the decisive implement that enables authoritarianism to take over.
We choose to take action&mdash;in light of all that is at stake&mdash;and build.
&nbsp;

Scale is proud to introduce our two Generative AI platforms – Donovan and EGP – to empower the U.S. government and enterprises to harness AI in a way that’s trusted, secure and moves at the speed of the world around us today.

A letter from our CEO | Scale AI

A letter from our CEO

Generative AI is capturing the imaginations of consumers and businesses alike. However, in this early era of adoption, most enterprises are yet to unlock true impact from this technology and are stuck in the experimental stages. Some are concerned about the security of their data and the reliability of model outputs. Others are overwhelmed by the many decisions to make regarding foundation models, prompt strategies, and tooling for developers.&nbsp;
Scale and <a href="https://www.anthropic.com/" target="_blank" rel="noopener">Anthropic</a> are excited to announce our partnership to bring to market a full-stack solution for Generative AI applications&nbsp;with a focus on safety and responsibility. This collaboration enables customers to bring Generative AI applications to production and to capture business value from this incredible technology.&nbsp;
Our partnership combines <a href="https://www.anthropic.com/product" target="_blank" rel="noopener">Anthropic&rsquo;s Claude</a> &ndash; a large language model guided by principles with the goal of creating outputs that are more consistent with human intent and values &ndash; with <a href="https://scale.com/generative-ai-platform">Scale&rsquo;s enterprise-grade platform</a> for app experimentation, development, and deployment.&nbsp;&nbsp;
Customers will be able to make the most of Claude with Scale&rsquo;s offerings:
<ul>
<li dir="ltr" aria-level="1">
Expert prompt engineering to guide users with the right prompt strategies and pre-built prompt templates, plus the ability to run experiments with different prompts.
</li>
<li dir="ltr" aria-level="1">
Data connectors to leverage proprietary data from databases and applications such as Confluence, Google Drive, Outlook, etc.
</li>
<li dir="ltr" aria-level="1">
Model validation through human evaluation to understand model blind spots and improve model performance, and features to continuously measure accuracy, brand alignment, and safety metrics, including sensitive content detection tailored to customers&rsquo; policies and brand.
</li>
<li dir="ltr" aria-level="1">
Enterprise-grade privacy and security, including deployment in a customer&rsquo;s AWS VPC and role-based access controls to ensure data and outputs are limited to the appropriate groups.
</li>
</ul>
Scale and Anthropic believe this represents a significant step forward for enterprises seeking to rapidly move beyond experimentation with Generative AI models. Designed to address concerns around data security, and model performance and reliability, this collaboration provides an accessible, &ldquo;all-in-one&rdquo; approach for customers to confidently tap into the power of Generative AI.

Scale and Anthropic are excited to announce our partnership to bring to market a full-stack solution for Generative AI applications, with a focus on safety and responsibility. This collaboration enables customers to bring Generative AI applications to production and to capture business value from this incredible technology. 

Scale + Anthropic: Full-Stack Generative AI for Enterprises | Scale AI

Scale + Anthropic: Full-Stack Generative AI for Enterprises

Scale AI, the data infrastructure for AI, has earned Impact Level 4 (IL-4) provisional authorization from the Defense Information Systems Agency (DISA), a significant milestone in Scale&rsquo;s ongoing commitment to providing the DoD with secure, industry-leading technology. This authorization allows Scale&rsquo;s cutting-edge machine learning (ML) platform and large language models (LLMs) access to defense applications with controlled unclassified information (CUI), such as personally identifying information, empowering the DoD with advanced AI capabilities to accelerate information accessibility. Scale is currently the only dedicated AI provider with IL-4 provisional authorization, underscoring its commitment to providing secure solutions for the U.S. government.&nbsp;
Scale&rsquo;s IL-4 provisional authorization expands AI accessibility for the DoD, allowing the agency to leverage Scale&rsquo;s Data Engine to curate and annotate data for a wide range of applications as well as Scale&rsquo;s Government AI platform for end-to-end solutions. Scale&rsquo;s powerful AI capabilities can empower analysts, operators, and leaders across the DoD with faster decision-making, improved operational efficiency, and accelerated innovation.
DISA, a combat support agency of the U.S. Department of Defense, is responsible for developing and maintaining the DoD Cloud Computing Security Requirements Guide (SRG). The Cloud Computing SRG defines the security requirements used by the DoD to assess the security posture of cloud service offerings, enabling cloud service providers (CSPs) to host DoD missions after receiving provisional authorization.&nbsp;
Scale&rsquo;s achievement of IL-4 provisional authorization demonstrates its adherence to rigorous security standards set forth by DISA and its commitment to supporting the DoD&rsquo;s mission-critical operations. With this achievement, Scale looks forward to unlocking the full potential of data and AI in the national security sector. For more information, please contact us at <a href="mailto:federal@scale.com">federal@scale.com</a>.&nbsp;
&nbsp;

Scale’s IL-4 provisional authorization expands AI accessibility for the DoD, allowing the agency to leverage Scale’s Data Engine to curate and annotate data for a wide range of applications as well as Scale’s Government AI platform for end-to-end solutions. 

Scale AI, the data infrastructure for AI, has earned Impact Level 4 (IL-4) provisional authorization from the Defense Information Systems Agency (DISA), a significant milestone in Scale’s ongoing commitment to providing the DoD with secure, industry-leading technology. 

Scale’s DISA IL-4 Provisional Authorization Increases Accessibility to AI for the Department of Defense (DoD) 

As we close out Women's History Month, we reflect on the incredible contributions of women throughout history and their importance in shaping our world today. In particular, we want to highlight the vital role that women have played and continue to play in STEM (Science, Technology, Engineering, and Math) fields.
Despite facing numerous challenges and barriers, women have made significant strides in STEM. From <a href="https://www.computerhistory.org/babbage/adalovelace/">Ada Lovelace</a>, the world's first computer programmer, to <a href="https://www.nasa.gov/content/katherine-johnson-biography">Katherine Johnson,</a> who helped put the first human on the moon, women have been pioneers in fields traditionally dominated by men. However, there is still much work to be done to address the gender gap in STEM, particularly in terms of representation and pay equity.
Throughout this month, Scale&rsquo;s Women ERG spoke to a few of its members to showcase the perspectives and experiences of women working in AI. We heard from Scaliens who have consistently broken barriers and achieved success in their fields, as well as those who have faced challenges and obstacles along the way.
We hope that by sharing these stories, we can help amplify the importance of diversity and inclusion in STEM and inspire the next generation of female leaders in the field. So, join us as we celebrate Women's History Month and recognize the incredible contributions of women in STEM. Together, let's continue to work towards a more equitable and inclusive future for all.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/503f7e9b-f8aa-4580-a096-2cba1e2402d6">
&ldquo;Having it all&rdquo; as a woman means different things to different women. I think this is important to acknowledge as I discuss what it means to me.
Last year, I began pursuing my MBA on top of my full-time job. When I first started, I quickly discovered that my definition of &ldquo;having it all&rdquo; was exhausting and unsustainable in my new season of life. It was time to change that definition.
Oftentimes women tend to take on more responsibilities both at work and at home. This was exactly what I was doing in every aspect of my life. It forced me to prioritize the limited free time I have. I had to set boundaries in my work and personal life to make room for school. I highly recommend The Book of Boundaries: Setting the Limits that Set You Free by Melissa Hartwig Urban as it was immensely helpful for me.
&ldquo;Having it all&rdquo; not only means different things to different women. It also means different things in different seasons of life. In my current season of life, &ldquo;having it all&rdquo; means shifting priorities and setting boundaries that help me succeed where I am today. When I finish my MBA, I&rsquo;ll enter a new season of life where &ldquo;having it all&rdquo; is going to evolve. And I can&rsquo;t wait to see what that looks like.&nbsp;
Yuri Maruyama
Head of Product Marketing
I joined Scale four and a half years ago when Scale was a forty-person company. At the time, I didn&rsquo;t think it would be possible for me to start a family while working at a hyper-growth, early-stage startup. It took a few years but I welcomed my first baby last summer. I was able to take and enjoy my full parental leave without getting pulled into any work or feeling any guilt.&nbsp;
Once I came back, I worried about how I would balance being back to work full-time while also being present for my family. I still often fall short of my own expectations, but to manage this, I&rsquo;ve blocked off my calendar in the mornings until 8:30am and in the evening between 5-8pm to spend with my family. Blocking this time and holding this boundary has helped to make me more present and productive at work during the 8:30am-5pm time when I am 100% focused on work. At the same time, I&rsquo;ve been able to be present for my family in the mornings and evenings without feeling guilty or like I need to check slack or email.
Bihan Jiang
Product Manager
Balancing work and life is both a challenge and a necessity. One way that I&rsquo;ve found balance is by scheduling time for activities that I enjoy, such as exploring coffee shops (I drink 3+ cups a day!), traveling, watching dramas, and playing board games like Catan and Tractor.
It&rsquo;s also critical to find passion in your work. As a product manager on the Spellbook team, I&rsquo;m responsible for driving the development of our product, collaborating with cross-functional teams, and staying on top of industry trends and customer needs. It can be challenging, but it&rsquo;s also incredibly exciting to be building in the generative AI space, which is evolving every day.
Achieving a balance between work and life is a continuous effort that requires prioritization. By making sure to allocate time for my hobbies and interests, as well as having a support system in place, I am able to maintain a strong sense of self while excelling in my career.
Karena Zhang
Account Executive
Passionate about living as a true &ldquo;Corporate Athlete,&rdquo; I find a myriad of ways to stay active and challenge myself in the office and beyond. When I first began my career, I was a Reformer Pilates instructor part-time while working full-time in tech. Now that I have since retired from being a fitness instructor, I have taken on a new challenge of traveling the world by way of sliding down mountains on wood planks (skiing). I was also a competitive college tennis player at Carnegie Mellon, so I am not new to navigating the stress of work and play, but I know how important it is to have a strong sense of self outside of work. I believe that no matter how stressful things get at work, I still have to live my life. Calendar blocks plus a passion for planning allow me to balance working hard, having fun, and making history.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f383b465-2282-480b-bdb2-246cb1198506">As a female engineer, I have experienced being overlooked or underestimated many times in my career. Most of the time, there's nobody to blame--these biases are just that deeply ingrained in everyone. Over time though, I have learned to use this to my advantage. When my coworkers had low expectations of me, as long as I still set ambitious goals for myself, I could make sure I blew their expectations out of the water. I constantly remind myself daily to be confident in myself and to not let my goals and abilities be dictated by the male-dominated environment I'm surrounded by. I also keep in mind that as a minority in the workplace, I can bring fresh perspectives to the table. Finding success in engineering definitely feels like an uphill battle sometimes, but that also makes success feel much more rewarding.
Iris Guo
Strategic Product Manager
Despite lots of excitement and eagerness, I didn&rsquo;t know what to expect coming into the role of a strategic product manager as a new grad. In the past, I had more traditional product manager internships at big tech companies where I was building software/ hardware features and focused on the user experiences. I haven&rsquo;t done operations before and this was my first time getting into the weeds of running a project as an STO. As a woman with high degrees of agreeableness and conscientiousness, I found it difficult to be the one calling the shots and not asking for approvals before execution. At first, I thought I had to come up with the solution for every problem, so I would put a lot of pressure on myself to do everything right and I wasn&rsquo;t used to asking for help since I didn&rsquo;t want to bother other people. I tend to be detail-oriented and diligent in my tasks, but sometimes I would miss out on the bigger picture for strategic thinking. When I was stressed out at work, I would lean into my passion for dancing as a creative channel for me to destress. Dancing helped me express my full sense of self and relieved my obsessive thoughts from work to stay present. I performed ballroom dancing growing up and I started learning k-pop/ jazz funk/ contemporary/ bachata/ salsa when I moved to SF six months ago. Every group dance required lots of teamwork to help each other to learn the harder part of the dance, and I learned to ask for help from people with more experiences, which I applied to problems at work. Through dancing, I also learned to experiment and iterate fast, because we would have different versions of the same dance and improve from trials and errors. Dancing enlightened me to see the bigger pictures and let go of the unimportant details with bold movements, which served as a constant reminder to me at work. I felt extremely grateful for having a healthy and mutually beneficial relationship between my hobby and my job, and I believe that everyone can find the right balance between work and life.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/5ac7d1f9-b882-43ac-9da9-8511d5062f95">I&rsquo;m thrilled to be a woman and working with other talented men and women here at Scale as a part of the AI revolution. As a society, we have a ways to go in terms of representation and equity across all STEM fields, and think that women have an especially important role to play in bringing diverse perspectives shaping AI products, and ensuring that the future of AI is just, equitable, and serves all of humanity better.

We hope that by sharing these stories, we can help amplify the importance of diversity and inclusion in STEM and inspire the next generation of female leaders in the field.

Throughout this month, Scale’s Women ERG spoke to a few of its members to showcase the perspectives and experiences of women working in AI.

Women’s History Month at Scale

Artificial Intelligence (AI) has been a game-changer for everyone. Together with data, AI has become a key component of a robust digital core&mdash;the main source of competitive advantage for companies today. The advent of&nbsp;<a href="https://www.accenture.com/us-en/insights/technology/generative-ai" data-analytics-link-name="generative ai" data-analytics-content-type="cta">generative AI</a>&nbsp;marks the beginning of a new era, one where businesses&nbsp;<a href="https://www.accenture.com/us-en/insights/consulting/total-enterprise-reinvention" data-analytics-link-name="reinvent" data-analytics-content-type="cta">reinvent</a>&nbsp;how they work. Just search for ChatGPT if you haven&rsquo;t already, and you&rsquo;ll get an idea of the possibilities. ChatGPT is derived from a large language model (LLM)&mdash;GPT4&mdash;developed by Open AI, which is a foundation model capable of generating human-like language and carrying out complex tasks across a number of functions.
We are entering an era of AI industrialization, where entire industries stand to be transformed by new, innovative applications of generative AI. One of the highest barriers to unlocking business value using generative AI and LLMs, however, is customizing and fine-tuning the pre-trained foundation models for their unique needs, using their own data. That&rsquo;s why Accenture is partnering with Scale AI.&nbsp;
<a title="Scale AI &amp; Accenture Partnership" href="https://scaleai.wistia.com/medias/2fjp89kejj">video</a>
&nbsp;
<h3 class="p1">Generative AI partner spotlight: Scale AI</h3>
Accenture&rsquo;s collaboration with&nbsp;<a href="https://scale.com/" target="_blank" rel="noopener" data-analytics-link-name="scale ai" data-analytics-content-type="cta">Scale AI</a>&nbsp;focuses on helping enterprises use their own data on general purpose models, customizing them to suit their business needs. According to CEO and founder Alexandr Wang, Scale AI has been &ldquo;quietly powering this generative AI wave, providing the data and infrastructure for many of the top companies to actually build large-scale foundation models, including Open AI.&rdquo;
Accenture and Scale AI are collaborating to provide an enterprise LLM platform for enterprise and government clients to fine-tune generative AI models that can learn from human input through a process called Reinforcement Learning from Human Feedback (RLHF). The result is generative models optimized for human interaction that are able to improve how people live and work.
For example, Scale AI was able to help retail clients gain higher shopping conversion and sales by using AI-generated product images for their newest products. And with social media clients, AI-generated content led to more than 100 million impressions.
Could humans have done this? Absolutely &ndash; but at what cost to the business?
This is business augmentation at the intersection of human and machine. By freeing up employees from automatable, routine tasks, we can redirect energy and focus to more strategic creative work that requires human intuition, decision-making and problem-solving skills - work that generative AI cannot do&mdash;at least not yet. The combination of human plus machine results in more efficient processes, improved decision-making and greater innovation.
Success with generative AI depends as much on people and training as it does on technology. It&rsquo;s up to business leaders to take the lead in upskilling employees and reshaping jobs and tasks to get the most out of technologies like generative AI.
Thanks to partnerships like this one with Scale AI, Accenture helps leaders create business value with generative AI technology quickly and at scale.

Scale AI + Accenture: Enhancing Business Value with Human Ingenuity

We&rsquo;re excited to announce that <a href="https://scale.com/spellbook">Spellbook</a> is now fully available for self-serve signups!&nbsp;
Spellbook is Scale&rsquo;s platform for large language model (LLM) apps. We believe the future of data science and machine learning is people building lightweight applications on top of large language models.&nbsp;
Speed and infrastructure quality is key&ndash;how easy is it to create a new LLM app, how fast can teams experiment, and how quickly can they get feedback on whether that experiment is better than a baseline or not? Once those experiments have signs of life, how efficiently can teams fine tune them, hone them with expert feedback, and launch them into production?&nbsp;
I&rsquo;ll walk through how you can create your own production-ready LLM app in less than 10 minutes. Follow along at <a href="https://spellbook.scale.com">https://spellbook.scale.com</a>!&nbsp;
App Creation
As an easy example, let&rsquo;s get started by creating a new marketing copy generation app that will take an event, size, and demographic, and generate an event description.&nbsp;
<figure class="image align-center"><img title="test" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/c23f338d-0dd8-492a-a458-70ff95c024fa" width="624" height="280">
<figcaption>App setup page in Spellbook</figcaption>
</figure>
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/2acb3b28-73c0-4bea-8ad3-385a5aba254a" width="580" height="423">
<figcaption>Example dataset with event, size, and demographic data.</figcaption>
</figure>
Quickly set up your prompt and see how your prompt runs on multiple rows of data.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/35ca21e5-3b43-4feb-ba8e-dd6d5215c8cb" width="3159" height="1599">
Evaluate Your LLM App
Make sure your model is outputting good generation with our human evaluation pipeline. Define evaluation criteria on how a human should determine whether your model output is good or bad. Here, we want to ensure our event description is relevant.
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/6ab0f3da-e7c8-4258-af87-b4455f71a35b" width="624" height="317">
<figcaption>Define human evaluation criteria for your model output.</figcaption>
</figure>
You&rsquo;ll get your results back in the form of a &ldquo;Hit rate&rdquo; along with downloadable results you can use to fine tune your model!
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/d9584b8a-0898-41a9-ac5c-70c343a7faff" width="624" height="319">
<figcaption>Receive evaluation results back after humans have finished reviewing.</figcaption>
</figure>
Deploying Your App
Once you have a model you&rsquo;re satisfied with, you can deploy your model to production in one click through our deployments support.&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/6d34a534-3432-4925-a3de-51dfef56ef71" width="624" height="315">
<figcaption>Create a new deployment for an App Variant.</figcaption>
</figure>
You&rsquo;ll receive an API endpoint that you can integrate directly into production, with built-in monitoring!&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/e8a2dfe8-b11c-4ecf-9d2d-30bbcc04ae57" width="624" height="315">
<figcaption>Integrate your LLM app into production using our code snippets, and monitor your requests and latency.</figcaption>
</figure>
As you improve your LLM apps, you can keep swapping them out in your deployments tab by selecting the most preferable variant. This ensures that once you&rsquo;ve gotten an endpoint integrated, you don&rsquo;t have to update your code again.&nbsp;
<figure class="image"><img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/ded46a41-1762-41f0-b7a2-732f2f2e420c" width="624" height="335">
<figcaption>Swap out your deployments with updated variants as you improve the quality of your LLM app.</figcaption>
</figure>
<h4 dir="ltr">Try Spellbook</h4>
As LLMs become more powerful, the ability to quickly deploy enterprise quality LLM apps to production and quickly iterate based on user feedback is crucial. Try <a href="http://spellbook.scale.com">Spellbook</a> today to create your first apps!&nbsp;
&nbsp;

We believe the future of data science and machine learning is people building lightweight applications on top of large language models. 

 Try Spellbook today to create your first apps! 

Create And Deploy LLM Apps in Less Than 10 Minutes

Getting data to train and evaluate your large language models has never been more Rapid! Introducing&hellip;Scale&rsquo;s Large Language Model Toolkit!&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/6b85a60e-339a-46fd-bcb7-ce485df871de" width="624" height="391">
We&rsquo;ve developed expedited workflows to help you set up projects within Rapid, Scale&rsquo;s self-serve data labeling tool. <a href="https://scale.com/rapid">Sign up</a> today to try it out and get $20 in free credits.
After signing up, you&rsquo;ll see all of our available project templates and our &ldquo;LLM Training Toolkit&rdquo; front and center. We&rsquo;ve divided our LLM toolkit into three buckets according to different workflows:&nbsp;
<ol>
<li dir="ltr" aria-level="1">
Demonstration Data: Collect human-generated text over various domains and use cases
</li>
<li dir="ltr" aria-level="1">
Comparison Data: Evaluate your model outputs with human feedback
</li>
<li dir="ltr" aria-level="1">
Trust &amp; Safety: Align your model to produce factually correct and safe content&nbsp;
</li>
</ol>
Demonstration Data&nbsp;
We have two subcategories of project types under the &ldquo;Demonstration Data&rdquo; bucket.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/09a3aa9a-f570-4ee7-93f8-42d2f47a9795" width="624" height="319">
(1) InstructGPT-style presets: Each of these comes with expert-written instructions, a preset taxonomy with the ability to add a customization section for your specific use case, the ability to upload your own prompts or have Scale create them for you, and a pre-trained workforce that has significant experience in these types of prompts. Check out Section 3.4 of the&nbsp;<a href="https://arxiv.org/pdf/2203.02155.pdf">InstructGPT paper</a> for yourself! 😉
(2) Demonstration corrections: If you have demonstrations or model outputs that need corrections, we can also assist you. Just upload your data or hook up your model endpoint and setup your taxonomy accordingly so that our labelers can get that done for you quickly.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/ef708a53-23de-49b1-ac2c-f0b744870c68">
Comparison Data&nbsp;
With our workflows, you can easily customize comparison annotation projects and quickly scale to production. You can ask a labeler to rank, rate on a Likert scale, and classify model outputs. You can either upload these model outputs or configure a model endpoint for labelers to interact with as they are making the comparisons.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/7eeab0fb-3f99-4b29-821e-5dcff793cfdb">
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/d27084b0-472c-417d-89e0-ea1364ef2e9f">
Trust &amp; Safety&nbsp;
Aligning your models with Rapid speed has never been easier. With our expertise and specialized workforce, we can work with you to identify problematic model output.&nbsp;
We support several use cases, including Grounded Generation, Red Teaming, Completion Citation, and Research-Based Correction.&nbsp;
<ul>
<li dir="ltr" aria-level="1">
Grounded Generation: Ask a labeler to research or read certain sources to verify uploaded model output.
</li>
<li dir="ltr" aria-level="1">
Red Teaming*: Provide an endpoint for labelers to chat with and elicit unsafe model outputs.&nbsp;
</li>
<li dir="ltr" aria-level="1">
Completion Citation*: Ask a labeler to attribute various phrases to sources that the information came from.&nbsp;
</li>
<li dir="ltr" aria-level="1">
Research-Based Correction*: Upload model output for labelers to find better sources for, edit/rewrite completions given any linked sources, add/edit/remove missing citations, etc.&nbsp;&nbsp;&nbsp;
</li>
</ul>
*Note: Use cases marked with * are currently not available via our self-serve platform. If you are interested, please contact sales to learn more.&nbsp;
For our Grounded Generation use case, we offer two self-serve, popular verification methods.
(1) Multi-class Classification: This involves selecting all error classes that pertain to the text via confidence-based consensus.&nbsp;
(2) Named Entity Recognition:&nbsp; This involves highlighting spans of erroneous text and indicating the relevant error types for each span.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/a35f53ff-4fae-48e1-b8f1-674fbcf3f752">
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/079d3f20-4345-4797-a4e3-31ba66781b0b"> And this is just the beginning! Our toolkit is constantly expanding, and we are excited to continuously be solving some of the world&rsquo;s hardest data annotation problems. Not seeing what you&rsquo;re looking for in the toolkit today or have other questions? Email us at: scale_rapid@scale.com.

Learn how to collect human-generated text, evaluate your model outputs with human feedback, and align your model to produce factually correct and safe content. 

Getting data to train and evaluate large language models (LLMs) has never been easier

Introducing: Scale’s Large Language Model Toolkit

<h3 dir="ltr">Introduction</h3>
OpenAI just released their new model that powers ChatGPT, GPT-3.5 Turbo. The best part &mdash; it's 10x cheaper than their previous best model.
OpenAI has found that GPT-3.5 Turbo is actually their best model for many non-chat use cases. Some early testers found it super easy to switch from the Davinci-003 model to GPT-3.5 Turbo with just a little bit of adjustment needed to their prompts. So, it's a much more cost-effective option for a lot of non-chat use cases.
Now that there is this new model available, it can be tough to know which one to use for your LLM apps in production. Spellbook makes it easy to build, compare, and deploy large language model apps, so you can evaluate GPT-3.5 Turbo against other models like Davinci-003 and FLAN to see which one performs best for your specific use case. In this blog post, we'll dive into the details of how these models differ on single-turn completions and what their strengths and weaknesses are, and share some real-world results from tests run on Spellbook- <a href="https://scale.com/spellbook">try it yourself today</a>!
<h3 dir="ltr">Classification&nbsp;</h3>
From our previous <a href="https://scale.com/blog/gpt-3-davinci-003-comparison">experiments</a>, we concluded that Davinci-003 performed much better than Davinci-002 with 0 shot prompts (92% vs 69.91%), and on par or slightly worse than Davinci-002 on few shot prompts (87.2% vs 91.6%). We observe a similar case with GPT-3.5 Turbo and Davinci-003.
GPT-3.5 Turbo performs better on 0 shot classification- on an insider trading classification example it achieves 91.72% accuracy, versus 82.02% for Davinci-003.
<figure class="image"><img src="https://lh4.googleusercontent.com/-qWbBYjvANJhwOpm25BihanYgq7goa1pw4QDINq3s6ZZedbwsHzuAOguit4S-2ug0TyVVx2DXgt7ep2J1MBiO_quaJrAWRlb4TRJJj-Te4zo5V3XHE58gNCbw_Rni-gFLx5gRxf0Aq0-bFM-aKUwN4c" width="624" height="349">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a 0-shot insider trading classification use case.</figcaption>
</figure>
However, Davinci-003 performs slightly better than GPT-3.5 Turbo with k-shot examples at 90.47% accuracy versus 88.37% for Davinci-003, but the quality is still lower than the GPT-3.5 Turbo 0 shot example.&nbsp;
<figure class="image"><img src="https://lh4.googleusercontent.com/iIefySacpNH98k4jEBgnUTSaRb5FCQiodLKt98KyvuI4-Z_iQp3lkH7bkQeKc-k-Sz2jnulr4wXT93pfU9visP46mOy0cfNXc3WVqr_b4M_meiDognTpgAL8vaHiB3cCW5eOCuhG_ZfEQPkSh6OWpYc" width="624" height="357">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on an insider trading classification use case with k-shot examples.</figcaption>
</figure>
The use of k-shot examples will likely not be very common with GPT-3.5 Turbo- including them in chat prompts can take up a significant amount of context, which is not ideal for a chatbot that receives a high volume of API requests. In cases where you&rsquo;re using GPT-3.5 Turbo for single-turn completion, this will be pretty much equivalent to adding k-shots for Davinci-003, but for multi-turn use cases, this can lead to inefficient resource usage if k-shots are included in every prompt.&nbsp;
On a more nuanced classification example, like sentiment analysis, we also see that GPT-3.5 Turbo outperforms Davinci-003 on 0 shot experiments (84.26% vs 78.57%).
<figure class="image"><img src="https://lh6.googleusercontent.com/ygoPuU_tuQoheEQ2N5Ut-7KG0dZjEap2zu6D-p4gijZ87fIfDz-zqnTY2oYaQIqWZhSMBrbrsxlGjZdFzogzJ1CZM5lgHmlsLnJkOpqSll81yjRYcioGG5BBK1xUraifOOiLojBPqj8t1HRZ7nBUwEk" width="624" height="360">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a 0-shot sentiment analysis use case.</figcaption>
</figure>
<h3 dir="ltr">Text Generation</h3>
If you're looking for an AI language model that can generate long, detailed answers to complex questions, GPT-3.5 Turbo might seem like the obvious choice since it's been trained on a massive dataset of human language and produces coherent, contextually-appropriate responses.
But what if you're looking for a model that can provide clear, concise responses? In that case, you might want to consider using Davinci-003 instead.
In our experiment, we provided both GPT-3.5 Turbo and Davinci-003 with a set of 30 questions, and asked each model to provide 30 answers. What we found was that GPT-3.5 Turbo&rsquo;s responses tended to be much longer than Davinci-003's, with many of them exceeding 100 words. In contrast, Davinci-003's responses were much more concise, often consisting of just a few sentences. Given a max tokens of 500, the average response from GPT-3.5 Turbo was 156 words (~208 tokens) nearly twice the amount of Davinci-003, whose responses were an average of 83 words (~111 tokens). 
<figure class="image"><img src="https://lh3.googleusercontent.com/XWVxJwRIeuE8s5I686NmO8WlR9DJPekjQ_TLSimXu8xm52eKrxOD0kJSyPcDqZj_mpvJ1aeA1soQ4DG8CYWsd3DqMnN4mE-SYj6Wqw7s5bu6XgdMYNQZjblaQcctic_hfJvUma8nqCEeitps7GeSucU" width="624" height="351">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a question answer use case.</figcaption>
</figure>
<h3 dir="ltr">Math&nbsp;</h3>
Notably, GPT-3.5 Turbo is significantly better than Davinci-003 at math. When evaluating by exact match with an answer key, GPT-3.5 Turbo gets the correct numerical answer 75% of the time, while Davinci-003 only gets 61% correct.
<figure class="image"><img src="https://lh6.googleusercontent.com/IG9V-yRgXR9vLTW0Myve1LV8Q8w1FpFPTs7t-YAaFXHAZQzRMUXZVvc64lb-2P6ERE79u_f_FY4eB1Y4dyK4kK59WtuEuMm-hNxQJ2C5m8J4ZuTxt1FE-C0ENR-iYK_8zXWId-SS7gy-qNIGeJ1ZGiY" width="624" height="313">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a math use case.</figcaption>
</figure>
We also selected a subset of the questions for expert human reviewers to evaluate the models correctness based on both the numerical correctness and the logical reasoning. Interestingly, upon asking both models to explain their answers, it became clear that the models would sometimes hallucinate reasoning for how they achieved the answer, even if it was numerically correct.&nbsp;
<figure class="image"><img src="https://lh4.googleusercontent.com/xmFdfVGuo3p81w5agMlTCOEuBLDkDdYHo6oniV8MYtIxc5IuqcFVti6vof3b3bb3dO-1QyOfyJ5ryVRleV6nSrPYpJc8w55vcRtnMY_KXhh7wB7HvNJ6ZB6bht9S8ijLJdgDviWDN6fJzXTrWCPQIJ8" width="624" height="315">
<figcaption>Evaluation configuration for human evaluation of math explanations.</figcaption>
</figure>
<figure class="image"><img src="https://lh4.googleusercontent.com/JRmEQBPW5OjYMxQWGH1ZyT7hVaeQ8dAUGx2Zy8qprMyCCwh1rrg1woFZzb1eSNS1sqkkgdI95cytMCljkoYVugA4tA3shO8SvP43hVQnOTW28Bkobru0sTHNqN6k1QXCjgwy9uSIcaPMIR4QxRdcBSg" width="624" height="229">
<figcaption>Quality drill down on human evaluations of math explanations from Davinci-003</figcaption>
</figure>
From the quality drill down summary from the completed tasks, Davinci-003 only got 44% of the questions correct. Repeating the same workflow for GPT-3.5 Turbo yielded 68% accuracy.&nbsp;
A couple interesting math lessons GPT-3.5 Turbo picked up that Davinci-003 struggled with:
<ul>
<li dir="ltr" aria-level="1">
Solving questions using the order of operations
</li>
</ul>
<figure class="image"><img src="https://lh5.googleusercontent.com/TLCynrXwIToi9u-tS5vFU-oBMcuKupJtA9foUG4ryegt3xwZCaGEhoPWmZpiteoUvTt8BzoRkVHqMY5L_xKOjwB2LLceaLxCNAHV0IYHJVLkVs9c7d8Cb6PnpVIMewge0AreLBYPPkBCRGqUL2MyMKg" width="624" height="276">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about order of operations.</figcaption>
</figure>
<ul>
<li dir="ltr" aria-level="1">
Plugging in variables into equations
</li>
</ul>
<figure class="image"><img src="https://lh3.googleusercontent.com/j63fVykVKFmDpt1qMIW3XPkntWiApDSvocH4oImJLdOr7quH0ESBTqDZwFSQpICgO1Idyja9iz_9UJn2cEDdOqB2z2Lv1UrRERGmb9qm9ojwBTuUwDhGBrvuZjsjeDpHYeFV2EM0sauHr3B_8shZNB0" width="624" height="147">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about plugging in variables into equations.</figcaption>
</figure>
<ul>
<li dir="ltr" aria-level="1">
The negative exponent rule
</li>
</ul>
<figure class="image"><img src="https://lh3.googleusercontent.com/sxzxXRaidO3EmcVfJE_fHvgUBwRfRkdpZ2RVhAtEO6R0tjjTdDLtueT0VtrPqDouqax6beTzb-d4m4Z3t8VwZaGopCOVj67sxPVCKzWKsloIgnx_PHftT6YsF0JEY2y3qfCn1bqp-vAILRwurbozbvc" width="624" height="212">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about negative exponents.</figcaption>
</figure>
<h3 dir="ltr">Base model swapping</h3>
It&rsquo;s worth noting that OpenAI has recommended the replacement of Davinci-003 implementations with the chat API in various instances, given its significantly lower cost of only $0.002 per 1,000 tokens. The ChatAPI can be used in a manner similar to the traditional Davinci-003 prompt format, where the system prompt is treated as the prompt body, the user message is treated as the input, and the assistant message serves as the completion.
For users who have LLM apps integrated in production using Spellbook, it&rsquo;s super easy to fork a variant, swap out the base model to GPT-3.5 Turbo and save it as a new one, and swap in the new variant, without having to write a single line of code.&nbsp;
<figure class="image"><img src="https://lh6.googleusercontent.com/Wp3bvT0IPiS0olJFhr-qiPieXViAuVBcq5q4ZAQOAcKJK8FFg2tOZp_nIsGcCdVZ0A8wdtdoAWj4CCoxMexDGD0IcUNyDPA-kFKvq0GuzHC3J_JWv4-T2oakkCKddvz13tS4OtmYzec0j-SXmLKOIws" width="624" height="325">
<figcaption>Quickly swap out your deployment&rsquo;s underlying variant to use GPT-3.5 Turbo without writing a single line of code.</figcaption>
</figure>
<h3 dir="ltr">Takeaways</h3>
OpenAI's new GPT-3.5 Turbo model offers a cost-effective option for many non-chat use cases. While GPT-3.5 Turbo performs well in 0-shot classification and math, Davinci-003 performs slightly better in k-shot classification and may be a better option for those looking for clear, concise responses that get straight to the point.
Spellbook makes it easy to build, compare, and deploy large language model apps, so you can evaluate GPT-3.5 Turbo against other models like Davinci-003 and FLAN to see which one performs best for your specific use case. With the ability to fork a variant and swap out the base model to GPT-3.5 Turbo, users can seamlessly integrate the new model into their LLM apps without writing any additional code.
Overall, GPT-3.5 Turbo performs better than Davinci-003 on most tasks at 10% of the cost - but is much more rambly. You can test out whether GPT-3.5 Turbo is better for your use case through <a href="https://scale.com/spellbook">Spellbook - sign up today</a>.

GPT-3.5 Turbo performs better than Davinci-003 on most tasks at 10% of the cost - but is much more rambly. You can test out whether GPT-3.5 Turbo is better for your use case through Spellbook

Compare the API behind ChatGPT to existing Davinci models using Spellbook

ChatGPT API: Cheaper, Faster, Wordier

I&rsquo;m not exactly sure of the moment in time and space in which I met Alex (at an event) several years ago. Given my background in the big data space, including at Apache Hadoop, Hortonworks, Cloudera etc. for more than a decade and a half I had no doubt in my mind that I had been at the forefront of the revolution that led enterprises to leverage massive amounts of data. At the time, the big data revolution had an unforeseeable impact.&nbsp;
Yet, Alex blew my mind and gave me a new perspective on data - for the emerging world of computer vision in the context of autonomous vehicles. Since then, Scale and Alex have been synonymous in my head with being at the very forefront of AI.
I was privileged to take a long break post my stint in the big data space. I&rsquo;ve wryly noted to friends that while I played my part in helping the Enterprise manage the entire lifecycle of data, I had a pole position in the entire corporate lifecycle too - from spinning out of Yahoo to found Hortonworks with my friends and being part of the team which built, at the time, the fastest enterprise software company to reach $100M in revenues, to going public in 3.5 years after founding, merging with its closest competitor, and taking the combined company private - the full circle.
I met Alex again at the end of last summer and was incredibly impressed with how far Scale had progressed. I had been working with a few startups and Alex kindly invited me to come over and pitch in at Scale - in any and every way. I was hooked immediately.&nbsp;
As a technologist, the work at Scale is obviously appealing - especially when aligned with the incredible team here with folks like Elliot, Russell, Vijay, Kate, Riley and many, many more. Like the rest of the world, I was absolutely enamored with the promise and the proximity of AI&rsquo;s value delivery. Much, much, more than big data, AI will impact every aspect of human endeavor and I could not believe my fortune in being this close to the opportunity epicenter to once again help enterprises realize value from this incredible technology. AI will impact our collective lives in every imaginable way, and in many ways that we lack the imagination to currently foresee.
So, at the end of my term straddling the new year, when Alex and the team at Scale offered me the chance to be a more permanent part of the company as Chief Product and Technology Officer - I absolutely jumped at the opportunity! I couldn&rsquo;t be more excited, and thankful!

Arun joined Scale as Chief Product & Technology Officer to accelerate the development of AI applications 

Why Arun Murthy joined Scale as Chief Product & Technology Officer

Have you ever listened to the song, &ldquo;Umi Says&rdquo; by Mos Def? It&rsquo;s a fantastic song and in the chorus, Mos Def sings, &ldquo;My Umi said, shine your light on the world. Shine your light for the world to see.&rdquo; My mother would say the same thing to me when I was younger and she still says it to this day! But what once started as rhythmic words of encouragement, now has begun to transform into a deeper, more raw sense of self. It has taught me that we can brandish one&rsquo;s individuality by sharing experiences, growing from adversities, and passing lessons learned onto my peers, and, to me, that is what I believe Black History Month is truly about. I never saw it as a design to cast blame or seek apologies; but rather a time to appreciate both Black culture and Black contributions. It is a time to develop a profound understanding of our current generation and those that came before us. Because from listening, comes understanding.&nbsp;&nbsp;
My family educated me on what being a Black person in America was like for them and what it may be like for me. They taught me how to navigate the waters and pushed me to hear others&rsquo; experiences when given the chance. And then in 2020, the George Floyd movement sparked discussion across the country and a societal shift began taking place. Various groups began broadcasting their desire for institutional change. Police training, workplace diversity, legislation geared towards discrimination; every facet that could touch bias or mistreatment within the Black community was under a microscope. Echoing a similar sentiment, different organizations and initiatives approached the problem in the form of open conversation. One of which was The Commit to C.A.R.E (Care About Racial Equity) Now initiative that was launched as a response to the consistent injustices, stereotypes, and negative attitudes about Black men in America. The initiative partnered with Dove Men+Care and the NBPA (National Basketball Player&rsquo;s Association) to create a portfolio focusing on &ldquo;celebrating the contributions of Black men in communities across the country and telling their stories as a part of our commitment to change the way Black men are seen and treated in our society.&rdquo; This initiative brought to light the perspectives and stories from CEOs, comedians, NBA players, authors, and actors alike to capture them in a series of videos and photo shoots. I was fortunate enough to participate in this collaboration and got to share my own experience on what it is like to be a Black man in America. These conversations moved the needle forward in the larger conversation about racial inequity and emphasized the need for tangible benchmarks for measuring this change.&nbsp;
&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh5.googleusercontent.com/MC-c6NkZ9xHbl5KiZg1PIjgUigQuZ3nlGGzPuxy1SvnmZ8T_EZtqqwpS6cRDQIDXOe6RFzqm4jupcz2sx0DBYtTKDBM-fiymx9WAqC2StoQbeYGfZFGUmDBdiuH1YoqRq4jXkqSXPAUr" width="235" height="235">
<figcaption>Here I am posing for the The Commit to C.A.R.E Now portfolio in 2020. This initiative partnered with Dove Men+Care and the National Basketball Players Association.</figcaption>
</figure>
Participating in this program and listening to the other men who joined me in it, taught me what it means to persevere and implement change. I heard their struggles, their hopes, and their own thoughts of what could be done to improve our culture&rsquo;s place in this world. Their views reaffirmed my belief: that when there is representation in any environment, avenues for change become achievable. At a granular level, having a diverse group of leaders can create networking opportunities for all communities through continuous involvement and influence. What were once one time donations, turns into institutionalized funding. Higher quality education becomes accessible rather than the faraway dream it used to be. These are examples of how equipping Black communities with the right tools can snowball into something that my community continually strives for: generational wealth in the form of capital, knowledge, and opportunity. Access to these tools allows for all generations to stand a chance against cronyism and social stratification.&nbsp;
While learning lessons from our current generation is powerful, I feel that it&rsquo;s important to also spotlight and commemorate the feats that Black Americans have accomplished in the past. Despite the history of racism and oppression, the Black community has paved the way for people like me. This community has been fighting for equality and fair opportunities so that Black people today may experience life in a way that they could not. They have demonstrated constant efforts to leave the world a better place than when they entered it and these very efforts have percolated from leaders like Martin Luther King Jr. down into the communities he served. As a result, these communities place a great deal of importance on sharing the impact that these historical endeavors have had on their lives by passing their history down to the next generation. So when the National Museum of African American History and Culture opened in 2016, my grandparents jumped at the opportunity to donate an artifact to the collection. Donating this artifact meant the world to my grandmother in particular because behind it, lies the history of her father.&nbsp;
Percival Leon Burrows was born in Barbados, and later in life, would move to Buffalo, NY and then Harlem to join the Universal Negro Improvement Association and African Communities League (UNIA-ACL) alongside Marcus Garvey. He then received a Black Star Line certificate for his service and passed it down to my grandmother who donated this very certificate to the museum. I&rsquo;ll never forget how proud my grandmother stood while looking at a small piece of my family&rsquo;s Black History on display in the nation&rsquo;s capital.&nbsp;
&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/Wlk_aR5NJZju3VJC7bnlsxxBPRXA7J1_Yw-EEctsfLTYUu65St43VRqht19euAUmP62SOC5QQ7p9tmtLlIi6_NMRc9SkF_CHmfvoxuhGF6LD09XNQZZz_DC8fe3PA8KKPoc8wTfAbrdC" width="246" height="207">
<figcaption>UNIA-ACL promoted the purchase of stock in this shipping company to encourage commercial trading among African people worldwide.</figcaption>
</figure>
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/Ke6ICNeyeAbtQLjJgR1MPS4-YxNIMJ0r7df5V3ba_ww1p8CsITwsEo_AMvEgOWXY3MzRajudXX2UhpCFCEkopzkTlh-sWUsDr8BV9ihsK-9h5HuGBulhydnXxVu3N8lyte2pN4_lWvWC" width="188" height="252">
<figcaption>Leota &ldquo;Burrows&rdquo; &amp; Edgar Henry pose in front of the Voices of Protest and Resistance exhibit at the National Museum of African American History and Culture.</figcaption>
</figure>
I am thankful for everything the Black community has done for me and I am especially grateful to my family and colleagues who have shaped the way I approach my work and my life. For without these efforts, I would not be in the position I am today. I would not have had the tools to become a Carnegie Mellon graduate. I would not have had the skills to play college ball and try out for the NBA&rsquo;s official minor league basketball organization. I would not have had the connections to intern under a U.S Senator. And I most definitely would not be a Security Engineer at Scale AI. But the efforts made by Commit to C.A.R.E Now, UNIA-ACL, and my grandmother, exhibit the lengths our community goes to progress the state of all African Americans and demonstrates what can happen when our community has a chance to shine their own light on the world. I hope that I am able to pay it forward with half the capacity that my community has done for me.&nbsp;
And Black history doesn&rsquo;t just stop in February! If you live in D.C like me and you&rsquo;re interested in learning more about the African American saga, I recommend visiting the <a href="https://nmaahc.si.edu/">National Museum of African American History and Culture</a>. It&rsquo;s packed full with extraordinary exhibits and thousands of artifacts that have been donated by people all over the world. If you aren&rsquo;t in the area but want to learn more, check out the Commit to C.A.R.E Now video (14:25)&nbsp;I was featured in! Or, if you use Instagram, checkout @blackarchives.co or @brownhistory.&nbsp;
<a href="https://www.youtube.com/watch?v=I82ioHNSEss">Commit to C.A.R.E Now video </a>
But for now, &ldquo;Alexa, play Umi Says, by Mos Def.&rdquo;
&nbsp;

Scale Celebrates 2023 Black History Month

Seth Henry shares his experiences as a Black man to celebrate Black History Month

Visions of Color

Scale hosted its first ever generative AI hackathon at our San Francisco headquarters on January 21st! We welcomed over 300 developers, engineers, and hackers from across the country to come in and create something awesome.&nbsp;
From side projects to chatbots, from work projects to generative art pieces &mdash; hackers submitted over 80 projects. The Hackathon was also buzzing with workshops where attendees could learn about topics like Advanced Prompt Engineering with&nbsp;<a href="https://twitter.com/goodside">Riley Goodside</a>, who helped attendees incorporate the art of large language model prompting into their projects.
Besides just meeting people and hacking, hackers could walk away with cool prizes, like win Napa tours, skydiving, drones and more.
<h2 dir="ltr" style="text-align: center;">Winners Feature&nbsp;</h2>
<h4 dir="ltr">First Place: GPT is All You Need for the Backend</h4>
Taking first place was a project called &ldquo;GPT is all you need for the backend&rdquo;. This innovative project automates backend processes with a Large Language Model (LLM). It can infer business logic based on the name of the API call and maintain state in the form of a JSON (lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate) blob. With this new tech, developers are now able to create an entire backend and database with no manual logic programming. Following the instructions, they can define their application and proceed to jump straight into making API calls.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/ck-TTZe5zkMITFldY26SHLQaImb5xCNsw4kury6by8yGjLBOfiLI7ct8VW5hjrfdjxIlrUPpyEvJaDeG_ww_rtbN1XECH501cxwgth3DyGWLYGLE1aEWtyZWWoKRRAycJMkXEja4mwIJCWwxbU7f5Jg" width="340" height="372">
<h4 dir="ltr">Second Place: Chempilot</h4>
&ldquo;Chempilot&rdquo;, the AI-enabled SMARTS (SMiles ARbitrary Target Specification - a chemical language used to describe molecular structures and substructures) analysis project, earned second place at the hackathon. Chempilot makes it easy to search chemical databases and guide generative models, by allowing users to generate SMARTS queries from natural language descriptions of molecules. This groundbreaking project was built by leveraging GPT3 and <a href="https://scale.com/spellbook">Spellbook</a>, while combining that with open-source datasets and libraries to build the dataset. Chempilot is incredibly useful for drug discovery, making it faster and easier than ever to find and analyze the right molecule. It&rsquo;s a unique combination of AI, chemistry, and cheminformatics expertise, making it an invaluable tool for any lab.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/P6PJGzsX8v9SKF7TTTRuUofrsaeXFPrIc_wZMnlVOELEE4IO8TNUFeE_Dw6uZfFAISPOw7qyQWG3LjwTqbLzANWcTlmEubMIwTbK9a-jHXFTyQwVL7k0DSTT8ol0bc9xEkpyN3dwRklNzn2wJ06ij_s" width="338" height="442">
The team behind the 'Rap Battles' project made waves with their use of generative AI to create rap battles, and won third place, as well as the Audience Choice award. Through the use of GPT-3 for lyrics, wavenet for vocals, and stable diffusion for the avatars, users can input the names of any two people and generate a rap battle between them. The team even took it one step further and launched the website on the day of the hackathon, providing a live demo of George Washington vs. Andrej Karpathy!
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/9XsTs_8nut6Kyi8GeRz_YcxhGgHSKyzbOy7Iz7K4K62zB128aEFl12sfkXLVZnDrlMdgoNOchwk26-PRp1UkhHkiExItA3y7dU_RZzubBN2oX-Rhmj_RjE1Y0NHeuI8PLnO255d0DVhueKc7zZdex18" width="470" height="385">
The project &ldquo;Node Fixer&rdquo;, won the Best Use of Spellbook award, and is the perfect tool for automating corrective action for syntax and runtime errors in JavaScript programs. With a combination of a VSCode extension, a rust binary, and Spellbook, Node Fixer allows developers to quickly identify and repair errors that cause their program to crash. When an error is detected, Node Fixer sends the error and the code to a Spellbook app, which then analyzes and produces a fix. The fixed code is written back into the editor, ensuring the developer can keep their program running smoothly and quickly. With Node Fixer, there is no need to manually fix trivial errors in your code.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/TjZZYOgCKD_RECp4Giu5cuB7TPqLWKdG22HokGYb1ovBnsfIi_UaWJG4zVasm8Q07HqkDWHI0BMsVrtPyj6RNXxwlJeLPQprqy4OvMBlSyXq3cVDjoy8L4zOkPEy2gSCyQvmcEHRG4DpspRFSDFpOnU" width="690" height="422">
<h3 dir="ltr" style="text-align: center;">Advanced Prompt Engineering with Riley Goodside&nbsp;</h3>
Riley Goodside also hosted a workshop on Advanced Prompt Engineering. Prompt engineering is the magic behind top-notch language model responses. A well-written prompt is crucial for avoiding off-topic, inconsistent, or offensive output. It's a challenging task that requires a nuanced understanding of how to talk to the models- there are research conferences and papers written on the best prompting techniques. The workshop garnered massive interest, and hackers filled up the office to learn the most innovative prompting strategies!
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/nbSgB9Mk6zXljYm7Rv4Y1i326SZzI3MTwQZUVFFUY0hD4Ifn6hLim_XqYbjTua-Vhoy35iAxvc54kv1TI4bXsiifFhrIEsP_L67WXo7AbecvahqxzvvMtRomljV2PdlNU0RuC4aC4E6D9TAQyMAVWQM" width="624" height="468">
Our hackathon was more than just a platform for hacking on cutting-edge projects, it was a community building event for passionate builders. You could satisfy your sweet tooth with Smitten's liquid nitrogen ice cream, bond with some furry friends at the puppy cuddling station, bring your own personal style to life with live screen printing on hoodies and tees, and even fuel up with a shot of espresso from the coffee bar. With so much to do and so many amazing people to meet, the hackathon was a prime opportunity for hackers to dive into generative AI and meet legends in the field.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/lx3H3DoC3k97GX7gLEe0Vsywhp5XR1c_TfRhIFQmzQgC2sqiyM98SP06fptcbPU_-iN7yr1gizlfWJq_kfz9BEZx3XbZJcPnJqvqLKd4ejyvdCfUhJVs08DtIfs3p-9PCDCbAOq6r9MSLV2FlGEpQ3Q" width="624" height="416"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh5.googleusercontent.com/UNaO30_derXRDPyGA2uFOjtJKkx9sVEb0J443_9eCnpu0L8_jXGQQ2S95dWAipTQ6wR6jJwPQMzUqF5vg2SqEFw8NWXTdVG407KhzsMk1GyqpZgHk8pcKGsvNF2TR6_0rfbyAjJw4OMbnEXBQyiKsgA" width="624" height="416">
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/SNojvekmg8mkVfpiAQPN7xpcQY7m6RQ0PEMmHTjueXISSRbPxs6AJ1RA6dE0el_6il7xa_3zy28GpfsQm8MvUV-g5VDxpMsb0m9b6LpjaiJgL4D8cYBk6EtV6hN8bXbXg3TpR2qMaI7StG8sLXJFxv8" width="624" height="416">
P.S. some of our favorite tweets about the hackathon :)&nbsp;
&nbsp;
<img src="https://lh4.googleusercontent.com/BNVKBDmYBj9GqY9w8ouGl2VFyr3SrK9dGf3w4I7L4WqZLQfAqiTDY_V_XB4BsBQrtauIE2lhnQ_idvzMIBIEG3eLOn4CEyXOhFUvHs7TLUHWp_PO36mWXeHXIfz5n6nGGoAlmwvSSVUa563UtBhkPcE" width="293" height="370"><img src="https://lh6.googleusercontent.com/BNTQxrxaDMWA94V4U_x5TqIfxvuQ_Gi4ZoF2EQndeFJ6dmun4V6NCr0MzCLWx6WdqTF07ylssd8M3Plr_1z34EaIuewbFVPiqI_bYODLxf7lklcOiiw42smrx_ihG5vp77VRpHrrGaZRz1okFRMZIlg" width="283" height="370">
<img src="https://lh4.googleusercontent.com/0ZcXCryx9vrkw9oyYu6zjT-wHLjFqM06sI11qIfgVAT7goE4tuw7qJZ6bY6sMbhgQUOxaFOvAT9QpDfM-4A4cF2Oiy_NMVSgGC85mtNK_Csq0sNny9l_WXCeIsjGUgML4rtYPgZYSb7QaKHzD_cjq-k" width="308" height="370"><img src="https://lh3.googleusercontent.com/qFWKetCapC-7aM4QbvTcu6lYU3_IWfgeo0RGYURuUO_7TONrtWEwbN21rtRcEDpN6tQzrLPr6UDkbACRlU1iQfZPKHcSKjH60zBiifS8csL47ULDQmqgOCraalVWh53zuhikF_WpJ5sMUww1MSsDNKI" width="256" height="370">
<img src="https://lh4.googleusercontent.com/XN5UpVoL4ue6mXgJozDI_6zX_zmAGKaMcZJKZDyjzd3jZQbU8ESShOMm3QTrIHQ-x6R4Fpcshpo-iwe43pjejP-QtE6uyj6DAcKKjm1yRMRpNQLFVzoQYsckpFR9pAG40scdDib3YeV8fZljGCyiMcg" width="301" height="370"><img src="https://lh3.googleusercontent.com/JgCQ71IhQOuqldXpvwLDY39GK95sgJJ1dNoaiCt107BCYSGk6TBa6Yg-ulJxXCD61J0B-CFlMvFbMYlhd0u7FdOU0UkT-4mpcjL7PfogbAVg2ZMyal7oMNzW4mkNTbqDczLJF1wQYRelKzIeyX5LJFU" width="283" height="370"><img src="https://lh4.googleusercontent.com/xUuNd3LblslS9aYrV8r-UWRNmvwbS28b7sMi-wgKiP5lGVWpHKEumgjARwwS05k9LZlOl7MQ2pjFopyGhCEqlZZrwlpAUgKnjzOowlpZb6XQUHtJnjdZh0Lj1SeelrXpBylEya3ZN4WXsB_DqOjIkgw" width="292" height="266"><img src="https://lh3.googleusercontent.com/crVzJnUXiAndcc-IcQ5kCFhMFHtuSIKEpQDFSMYGJFl_11XRI2Paw2EyAnIT5zQ9bnGyWyB1cVypEW-VEpQdLy6XqAgtDMQbs2GeRU1YRiVf6d9X-gvmjeFO9kEuVLyr3Qg45VxrbgTvQjwfERhDTIs" width="299" height="266"><img src="https://lh3.googleusercontent.com/snJGc791L74WwKlDcAVtlj4mYw7XdA82nRWRViQLSDaNuzLNPp75sqJ6bBuCvX0JBEmPXmm5AT8lQtiyJeMBZ7TzDV5_Q4EF5OPak51gQUqZfHvlkfite3JIAggOrnmnNl2L8SFmUUyTe0sY8TYFL3c" width="292" height="249"><img src="https://lh4.googleusercontent.com/De-fqScHljBxUGhJJSoEVwZTHLO0qojNg0MfvKDPQH2Qe8Hzr7KB9o4D3Xr6-296aZOsQ_PhC2TO_6tSqLcdUsJRRCfVdIMHMiwk_l269wxsNttxpeC7eObiU0JUt782CzMLgiuZa1xgt24p8a-XjGM" width="297" height="249">
&nbsp;
&nbsp;
&nbsp;

The hackathon was more than just a platform for hacking on cutting-edge projects, it was a community building event for passionate builders.

Scale hosted it's first ever generative AI hackathon

Scale's Generative AI Hackathon

In order to facilitate the automation and speed of maintaining surface to air missiles (SAM) systems chain of custody, Scale developed an AI powered system to track movements at these sites using EO satellite imagery. This blog post explores some of the implementation and novel concepts around the problem.
SAM systems are guided missiles that are typically launched from a mobile launch vehicle or a stationary launcher and used to defend against air attacks. Two common stationary SAM systems are the SA-2 and SA-3, often used in former Soviet satellite states.&nbsp;
From overhead imagery, these systems are typically identified by hexagonal patterns in the ground with launchers placed around the hexagon (with or without missiles), a radar placed near the center (typically accompanied by a communication vehicle), and missile transloader vehicles dispersed throughout the site.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1efbb734-34e8-484f-8d41-f0858d34f69a" alt="" width="1200" height="720">
<h6 dir="ltr" style="text-align: right;"><code class="c-mrkdwn__code" data-stringify-type="code">Image Source: Skyfi</code>&nbsp;</h6>
A SAM system's perceived threat level is defined by the activity over time at a given site and the objects of interest at the site. For example, a site with ground vehicles and launchers present may be considered occupied but not operationally functional and therefore less of a threat. However, if a radar were to show up at that site it could then become operational, which is a potential cause for alert.&nbsp;
<h2 dir="ltr">Machine Learning Approach</h2>
<h3 dir="ltr">Narrowing the problem set</h3>
Tracking the status of SAM systems requires maintaining custody of 4 object classes: radars, ground vehicles, launchers, and missiles. However, there exists nuance about how these objects are detected: SA-2/SA-3 sites can be located in both dense urban and rural settings. Training an object detection model on a general region with a SAM system will lead to perceived annotation errors and false positives.&nbsp;
If, for example, there is a parking lot nearby (as there is in the image above) that does not actually contain vehicles relevant to the status of the SAM site, feeding this parking lot into the neural network without any annotations will likely confuse training and lead to declined model performance. Similarly, during inference time, the network is likely to output a number of false positive predictions around the vehicles in this parking lot when we are only interested in military vehicles relevant and directly within the SAM site.&nbsp;&nbsp;
To counteract this, Scale built a general SAM site object detector which places a focused bounding box around the site itself (see algorithm A in the figure below). This allows preprocessing for cropping out the specific site itself to feed into the final object detection network (see algorithm B in the figure below). Additionally, it provides a stand alone algorithm that can be used for broad area searches of new SA-2/SA-3 sites around the world.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/0c0b67e2-15fb-4f41-a1c0-c6b0fdb0d5f4" alt="" width="1200" height="430">
<h3 dir="ltr">Including Apriori Knowledge</h3>
Custody tracking in satellite images provides a unique training advantage over a static, stand alone object detection network that intakes a raw image and outputs detection predictions. In the case of custody tracking, there is almost always a prior timepoint of that Area of Interest that has been reviewed by an intel analyst and &ldquo;labeled&rdquo; with points. The work of the intel analyst can provide incredibly valuable information to the neural network learning to detect these objects on its own. To take advantage of this, instead of just feeding in an image of a SAM site to the network, we feed in both the image of the SAM site and a heatmap of the location of the objects of interest from the prior timepoint.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/46203af2-8fdd-4bbd-b10a-7a3c07c10497" alt="" width="1200" height="888">
The addition of these prior heatmaps improved algorithm detection performance across all 4 classes of interest. Object detection performance on these classes is measured using mean average precision at 50% IoU (mAP50).
<div dir="ltr" align="center">
<table><colgroup><col width="218"><col width="277"></colgroup>
<tbody>
<tr>
<td>
Class of interest
</td>
<td>
mAP50% improvement w/ Prior Heatmaps
</td>
</tr>
<tr>
<td>
SAM specific ground vehicles
</td>
<td>
4.2%
</td>
</tr>
<tr>
<td>
Launchers
</td>
<td>
17%
</td>
</tr>
<tr>
<td>
Radars
</td>
<td>
20.9%
</td>
</tr>
<tr>
<td>
Missiles
</td>
<td>
13%
</td>
</tr>
</tbody>
</table>
</div>
<h2 dir="ltr">Conclusion</h2>
The simultaneous transition to great power competition and explosion of imagery data creates an urgent need for imagery analysts to exploit imagery faster than ever before. High performance SAM detection algorithms allow analysts to receive automated disposition of forces insights at machine-speed, freeing analysts to spend their time on more complex analyses. Because the algorithm&rsquo;s outputs are structured data, analysts can further automate downstream processes such as setting up alerts for anomalous behavior or maintaining chain of custody.&nbsp;
At Scale, we believe technology should be used to empower (not replace) the analyst in crafting their narratives to deliver decision advantage.&nbsp;
NOTE:
This material is based upon work supported by the USAF AFRL/SBRK under Contract No. FA864921P0899. Any opinions, findings and conclusions or recommendations expressed in this material are those of&nbsp; the author(s) and do not necessarily reflect the views of the USAF AFRL/SBRK.&nbsp;
&nbsp;

High performance SAM detection algorithms allow analysts to receive automated disposition of forces insights at machine-speed

Maintaining (SAM) Systems

SAM Introduction

How Large Language Models are Trained and Tuned using Reinforcement Learning with Human Feedback (RLHF).

Why is ChatGPT so good?

ChatGPT has captured LLM headlines and amazed the AI community with its extensive natural language processing capabilities. A new LLM from Anthropic called Claude is competitive with ChatGPT and offers great promise. We evaluate both models head to head and give you our thoughts on how they compare.

Claude vs. ChatGPT: Let’s Think Step by Step

Meet Claude: Anthropic’s Rival to ChatGPT

Scale AI, the data infrastructure for AI, has been named a Cool Vendor in the 2022 <a href="https://www.gartner.com/doc/reprints?id=1-2C7YMC4Z&amp;ct=230109&amp;st=sb">Gartner Cool Vendors&trade; in Data-Centric AI</a> by Svetlana Sicular, Arun Chandrasekaran, Sumit Agarwal, Oleksandr Matvitskyy and Anthony Mullen.
The biggest barriers for organizations looking to implement data-centric AI strategies are having the necessary resources, talent, experience and high-quality data needed to feed algorithms and leverage the power of AI. Scale provides AI solutions tailored to business use cases and machine learning tools to unlock breakthroughs like Generative AI and operationalize AI for all organizations spanning the U.S. government, researchers, startups and Fortune 500 companies across e-commerce, logistics, technology, fintech, and more industries. Organizations such as Microsoft, Open AI, SAP, Flexport and the U.S. Army partner with Scale to solve problems with data labeling and annotation, scenario-based model testing and validation, ​​content understanding and contextualization, AI catalog for asset reusability and more.
According to Gartner, &ldquo;Data challenges plague AI implementations. They are often unexpected because AI teams focus on algorithms, and data concerns come as an afterthought.&rdquo; It recommends that when evaluating data-centric AI solutions, organizations should &ldquo;work with stakeholders to determine what data characteristics are high value for AI,&rdquo; and &ldquo;add AI capabilities such as AI-specific data management, synthetic data and data labeling &ndash; to your data management toolbox.&rdquo;
&ldquo;It&rsquo;s an honor to be named a Data-Centric AI Cool Vendor by Gartner. We believe that it is a testament to Scale&rsquo;s mission to accelerate the development of AI applications and showcases the market impact of our vision,&rdquo; said Alexandr Wang, CEO and Founder of Scale AI. &ldquo;The solutions that we provide enable organizations to reap the benefits of AI, whether they are building or applying AI. This means better business outcomes, a real return on investment, and a faster time to production.&rdquo;
According to the Gartner report, &ldquo;By 2025, 90% of new data and analytics deployments will be through an established data ecosystem, causing consolidation across the data and analytics market.&rdquo; Therefore, introducing a data management approach, such as data quality, data diversity, data and analytics governance, and data literacy to AI teams now, is imperative to any organization's AI strategy.
Download the full report <a href="https://www.gartner.com/doc/reprints?id=1-2C7YMC4Z&amp;ct=230109&amp;st=sb">HERE</a>.
Gartner Disclaimer
GARTNER and COOL VENDORS are the registered trademarks of Gartner Inc., and/or its affiliates and has been used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner&rsquo;s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
About Scale AI
Scale accelerates AI development by providing the data infrastructure for AI. Its data-centric platform manages the entire ML lifecycle, from data annotation and curation to model testing and evaluation, enabling any organization &ndash; from the world&rsquo;s most advanced AI teams to legacy organizations &ndash; to develop and deploy impactful AI solutions. Scale combines ML technology with skilled human insight to ensure every AI application is built on a foundation of high-quality ground truth data. Scale is trusted by the largest technology companies in the world, including Microsoft, Toyota, Nuro, Samsung, Square, PayPal, SAP, Instacart, Pinterest, Brex, Flexport and U.S. government agencies like the U.S. Army and Air Force. Founded in 2016 and valued at $7.3 billion, Scale is backed by Founders Fund, Accel, Index, and Tiger Global.

Scale AI, the data infrastructure for AI, has been named a Cool Vendor in the 2022 Gartner Cool Vendors™ in Data-Centric AI.

Scale AI Named a 2022 Cool Vendor by Gartner® | Scale AI

Scale AI Named a 2022 Cool Vendor by Gartner®

<div dir="ltr">&nbsp;</div>
<div dir="ltr">Today I have to announce the hardest change I&rsquo;ve ever had to make at Scale. I have made the difficult decision to reduce the size of our team by 20%, which means saying goodbye to many talented Scaliens. If you are among those impacted, you will be contacted shortly with further details via your personal email as well as offered time for a 1:1 conversation with a manager today. This was not a decision made lightly, and it&rsquo;s one of many steps we are taking in order to ensure Scale is operating responsibly for the long-term health and success of the business. I know that this is tough news for everyone, especially those impacted, and you likely have many questions, the most pressing of which I aim to answer for you now:</div>
<div dir="ltr">&nbsp;</div>
<div dir="ltr">How did we get here? I take full responsibility for the decisions that have led us to this point. Over the past several years, interest from enterprises and governments in AI has grown rapidly. As a result, I made the decision to grow the team aggressively in order to take advantage of what I thought was our new normal. For a time, this seemed to prove out&mdash;we saw strong sales growth through 2021 and 2022. As a result, we increased headcount assuming the massive growth would continue. However, the macro environment has changed dramatically in recent quarters, which is something I failed to predict. Many of the industries we serve, such as e-commerce and consumer technology, have been buoyed by the pandemic and are now experiencing a painful market correction. As a result, we need to prepare ourselves for a very different economic environment. Given the uncertainty many of the industries we serve face, when I re-assessed our investment level against these market realities, it became clear that we needed to realign our investment to adjust to this new environment. While many other companies have made similarly difficult decisions recently, we spent months looking for ways to avoid it, but unfortunately we came to the conclusion that we needed to make these changes as well.</div>
<div dir="ltr">&nbsp;</div>
<div dir="ltr">How are departing employees being supported? We want to be as respectful and supportive through this transition as possible. Some of the core details include: 
<ul>
<li>Severance pay:&nbsp;We will pay the equivalent of at least 8 weeks of severance for departing Scaliens, with additional severance for those with longer tenure.</li>
<li>Healthcare:&nbsp;We&rsquo;ll pay the cash equivalent of 3 months of existing healthcare premiums for U.S. Scaliens.</li>
<li>Equity:&nbsp;For those with equity, we&rsquo;re waiving the 1-year equity cliff for Scaliens with less than 1 year of tenure. We will also offer a 3-year exercise window for Scaliens with less than 2 years of tenure.</li>
<li>Additional support:&nbsp;We&rsquo;ll do our best to connect departing Scaliens with other companies and offer resume reviews with our recruiting leads. We will also provide immigration support for Scaliens on visas that require continued employment.</li>
</ul>
<div>What other changes are we making? We have looked across the company at ways to reduce spend and preserve as many roles as possible. As part of planning we have set significant operating expense reduction goals for each function, adjusted our hiring targets, and are re-assessing any new offices. We need everyone to do their part to stay cost-conscious and efficient when it comes to spending throughout the year. What does this mean for Scale&rsquo;s future? While these changes are painful, Scale continues to be well-positioned within the undeniably consequential AI industry. Our technology has made massive strides in 2022, and the promise for AI to impact every industry has never been greater. At the same time, the core tenet upon which Scale was founded&mdash;that powerful AI systems are dependent on high quality data&mdash;still remains true. Each of the big AI breakthroughs (ChatGPT, GPT-3, DALL-E) have been made possible through extensive data collection and annotation infrastructure. The importance of our work will only magnify as our customers look to build AI systems that are more reliable, accurate, and powerful. It has never been clearer that AI will transform every industry. Our national security work with the US and allied governments in building AI capabilities is critical, and we don&rsquo;t expect that criticality to waver. We believe automation and AI will become a core focus area in this new environment. These themes continue to play out in the data we see&mdash;our growth rates are strong, we&rsquo;ve seen tremendous early adoption of our newly launched Generative AI products, and our customers are continuing to see impactful results leveraging AI. In the week ahead, we&rsquo;ll share more on our strategy for this year and beyond, and how our remaining team and streamlined operations will help achieve this. In the meantime, I want to extend my gratitude to every single Scalien who is leaving us today. Your contributions have helped move the world of AI forward.</div>
</div>

Company Update

Software engineers with no domain expertise can now access a trained and deployed model with just a single click after image data upload.

Access Trained Models Based on Your Uploaded Image Data Right in Rapid

Models are Now Available in Rapid

On November 28th, OpenAI released a new addition to the GPT-3 model family: <code>davinci-003</code>. This latest model builds on <a href="https://openai.com/blog/instruction-following/" target="_blank" rel="noopener noreferrer">InstructGPT</a>, using reinforcement learning with human feedback to better align language models with human instructions. Unlike <code>davinci-002</code>, which uses supervised fine-tuning on human-written demonstrations and highly scored model samples to improve generation quality, <code>davinci-003</code> is a true reinforcement learning with human feedback (RLHF) model. It is trained with <a href="https://openai.com/blog/openai-baselines-ppo/" target="_blank" rel="noopener noreferrer">PPO</a> to optimize the generated text&rsquo;s score against a separate &ldquo;reward model&rdquo;, which is trained on rating comparisons by human graders of different model outputs. More details can be found in OpenAI&rsquo;s <a href="https://beta.openai.com/docs/model-index-for-researchers" target="_blank" rel="noopener noreferrer">model index for researchers</a>. The net result is that <code>davinci-003</code> is tuned to produce outputs that it thinks would be scored highly by humans. Scale is proud to partner with OpenAI to provide this human feedback.
OpenAI&rsquo;s announcement email mentions the following improvements for <code>davinci-003</code>:
<ul>
<li>&ldquo;It produces higher quality writing. This will help your applications deliver clearer, more engaging, and more compelling content.</li>
<li>It can handle more complex instructions, meaning you can get even more creative with how you make use of its capabilities now.</li>
<li>It&rsquo;s better at longer form content generation, allowing you to take on tasks that would have previously been too difficult to achieve.&rdquo;</li>
</ul>
But how much better is <code>davinci-003</code>, really? We decided to put it to the test quantitatively. Using <a href="https://scale.com/spellbook" target="_blank" rel="noopener noreferrer">Scale Spellbook</a>, the platform to build, compare and deploy large language model apps, we evaluated <code>davinci-003</code> versus <code>davinci-002</code> across tasks ranging from few- and zero-shot classification, summarization, and poetry writing. Here&rsquo;s what we found.
<h3>Classification</h3>
It&rsquo;s well known that <code>davinci-002</code> performs very well on classification with a few-shot prompt, and it appears that <code>davinci-003</code> offers comparable (but slightly worse) performance few-shot. Using Spellbook&rsquo;s built-in evaluation feature on 250 Yelp reviews, we compare the two models&rsquo; classification of these reviews as Positive, Negative, or Neutral in a four-shot prompt&mdash;meaning four labeled examples are included in the prompt itself.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F65f8dfd9c9744c15967d03341c5cdc73">
The davinci-003 model has 87.2% accuracy on this 250 Yelp review dataset sample, slightly lower than davinci-002&rsquo;s 91.6%.
However, if we ask the two models to classify these same reviews in a zero-shot prompt, we find that <code>davinci-003</code> scores more than 30% higher in accuracy than <code>davinci-002</code>&mdash;and on par with a <code>davinci-002</code> few-shot prompt! This is a substantial improvement in zero-shot classification performance. In this classification setting, adding examples to the prompt actually lowers <code>davinci-003</code>&rsquo;s accuracy.
<img src="https://lh6.googleusercontent.com/Wr30hJNdDAekBQTxtBRBN40_GR1CIElZlygaJSrAbGd6OK20V-pGS_bI5BoCF2SS4az3z7EFPYde3k5TnoVKHZd-68dwlhtHctxB7XtW5zqOXOjQkY2P8SxCkAjzlmg1PpKQ1hzEgob2W18laWD8FfSjNYShqjurevSldtMENyh8DUALgcMeYi4HQKi-">
Beyond accuracy, we found that asking <code>davinci-003</code> to explain its rationale in a zero-shot prompt works reliably, and provides a better explanation than a one-shot <code>davinci-002</code> about 75% of the time, based on Scale&rsquo;s human evaluation workforce judging a sample of 100 outputs.
<h3>Text Generation</h3>
The most notable performance leap comes in how <code>davinci-003</code> understands word and sentence structure, especially rhyming and syllables. In prompting for a poem that rhymes, <code>davinci-003</code> constructs a desired output, whereas <code>davinci-002</code> largely fails to implement rhyming.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F31cb06da7e1a488a91f62ef14feee823">
<code>davinci-003</code> is also better equipped to handle requests for complex output structures, such as iambic pentameter (think Shakespeare). <code>davinci-002</code> certainly provides a poem, but ignores the request for lines of 5 pairs of stressed and unstressed syllables (buh-BUM). While not perfect, <code>davinci-003</code>&rsquo;s output is significantly more faithful to the requested prompt.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fc3fc8edf1aea432bb7bb5118a6dc8c82">
<h3>Summarization</h3>
One practical difference between the two variants is verbosity. In our experimentation, <code>davinci-003</code> consistently generates longer outputs than <code>davinci-002</code> on the same prompts. This is most visible in summarization tasks, where the value of the result may depend on its brevity.
In a sample of 60 articles, we tested both variants&rsquo; summarization capabilities by asking them to summarize the introductory paragraphs of lengthy Wikipedia pages. The additional text generated by <code>davinci-003</code> is typically beneficial, including information central to an article&rsquo;s importance. However, the resulting summaries are also wordy. In our analysis, <code>davinci-003</code> generates outputs about 65% longer than <code>davinci-002</code> under identical prompts.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fbbd0ed222dff4216ab9173d07d176890">
It will be important for GPT-3 users to bear in mind <code>davinci-003</code>&rsquo;s talkativeness when choosing between the two variants. In our testing, given the task of generating a 1-sentence summary, <code>davinci-003</code> produced 42 words on average, nearly doubling <code>davinci-002</code>&lsquo;s average of 23. The shortest summary <code>davinci-003</code> gave back was a beefy 32 words, compared to only 12 words for <code>davinci-002</code>. This tendency extends to multi-sentence summaries as well.
For use cases that value brevity, <code>davinci-002</code> is the safer choice. Nevertheless, for many use cases, the longer and more detailed completions provided by <code>davinci-003</code> can be beneficial.&nbsp;
<h3>Conclusion</h3>
Overall, <code>davinci-003</code> is a meaningful improvement from <code>davinci-002</code>, especially for zero-shot classification and complex text generation.
While many of the benefits of the latest iteration of GPT-3 are apparent at the surface, concrete evaluation metrics ensure models and prompt strategies are performing as well as possible.
Luckily, Spellbook makes that easy, allowing users to run instant programmatic evaluation in classification use cases, or sending model outputs to Scale&rsquo;s high-quality workforce to get quantitative evaluation metrics for generative use cases.
<a href="https://scale.com/spellbook" target="_blank" rel="noopener noreferrer">Sign up for Spellbook today!</a>
P.S. If you&rsquo;re looking for a little dramatic flair in your emails to coworkers, <code>davinci-003</code> has you covered. It performs significantly better at creatively translating text or even explaining code as different celebrities in a zero-shot setting.
&nbsp;
BranchCommitmaintestmaintest <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa4f4e03e1cef4d28a44fcf465150aeca">
<pre class="ql-syntax" spellcheck="false"> Write a short rhyming poem about winter in New York davinci-002
davinci-003
</pre>
In New York City When winter descends
The air is so crisp and chilly
And the wind never ends
The snowflakes are so big and fluffy
You just want to grab a handful
And then make a snowball
People hurry to and fro
Bundled up in coats
Hats, and scarves, and gloves
No one wants to be caught
Lights are twinkling in the night, It's wintertime in New York City so bright.
The snow is glistening, the wind is cold,
The atmosphere is really bold.
The snowflakes fall with a gentle grace,
Adorning the city with a lovely lace.
The air is crisp, and the people merry,
It's a magical winter in New York City.
Text Generation
The most notable performance leap comes in how davinci-003 understands word and sentence structure, especially rhyming and syllables. In prompting for a poem that rhymes, davinci-003 constructs a desired output, whereas davinci-002 largely fails to implement rhyming.

On November 28th, OpenAI released a new addition to the GPT-3 model family:
 <code>davinci-003</code>. This latest model builds on
 <a
 href="https://openai.com/blog/instruction-following/"
 rel="noopener noreferrer"
 target="_blank"
 >InstructGPT</a
 >, using reinforcement learning with human feedback to better align language
 models with human instructions. Unlike <code>davinci-002</code>, which uses
 supervised fine-tuning on human-written demonstrations and highly scored model
 samples to improve generation quality, <code>davinci-003</code> is a true
 reinforcement learning with human feedback (RLHF) model. It is trained with
 <a
 href="https://openai.com/blog/openai-baselines-ppo/"
 rel="noopener noreferrer"
 target="_blank"
 >PPO</a
 >
 to optimize the generated text’s score against a separate “reward model”,
 which is trained on rating comparisons by human graders of different model
 outputs. More details can be found in OpenAI’s
 <a
 href="https://beta.openai.com/docs/model-index-for-researchers"
 rel="noopener noreferrer"
 target="_blank"
 >model index for researchers</a
 >. The net result is that <code>davinci-003</code> is tuned to produce outputs
 that it thinks would be scored highly by humans. Scale is proud to partner
 with OpenAI to provide this human feedback.


 OpenAI’s announcement email mentions the following improvements for
 <code>davinci-003</code>:

<ul>
 <li>
 “It produces higher quality writing. This will help your applications
 deliver clearer, more engaging, and more compelling content.
 </li>
 <li>
 It can handle more complex instructions, meaning you can get even more
 creative with how you make use of its capabilities now.
 </li>
 <li>
 It’s better at longer form content generation, allowing you to take on tasks
 that would have previously been too difficult to achieve.”
 </li>
</ul>

 But how much better is <code>davinci-003</code>, really? We decided to put it
 to the test quantitatively. Using
 <a
 href="https://scale.com/spellbook"
 rel="noopener noreferrer"
 target="_blank"
 >Scale Spellbook</a
 >, the platform to build, compare and deploy large language model apps, we
 evaluated <code>davinci-003</code> versus <code>davinci-002</code> across
 tasks ranging from few- and zero-shot classification, summarization, and
 poetry writing. Here’s what we found.

<h3>Classification</h3>

 It’s well known that <code>davinci-002</code> performs very well on
 classification with a few-shot prompt, and it appears that
 <code>davinci-003</code> offers comparable (but slightly worse) performance
 few-shot. Using Spellbook’s built-in evaluation feature on 250 Yelp reviews,
 we compare the two models’ classification of these reviews as Positive,
 Negative, or Neutral in a four-shot prompt—meaning four labeled examples are
 included in the prompt itself.


 <img
 src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F65f8dfd9c9744c15967d03341c5cdc73"
 />


 The davinci-003 model has 87.2% accuracy on this 250 Yelp review dataset
 sample, slightly lower than davinci-002’s 91.6%.


 However, if we ask the two models to classify these same reviews in a
 zero-shot prompt, we find that <code>davinci-003</code> scores more than 30%
 higher in accuracy than <code>davinci-002</code>—and on par with a
 <code>davinci-002</code> few-shot prompt! This is a substantial improvement in
 zero-shot classification performance. In this classification setting, adding
 examples to the prompt actually lowers <code>davinci-003</code>’s accuracy.


 <img
 src="https://lh6.googleusercontent.com/Wr30hJNdDAekBQTxtBRBN40_GR1CIElZlygaJSrAbGd6OK20V-pGS_bI5BoCF2SS4az3z7EFPYde3k5TnoVKHZd-68dwlhtHctxB7XtW5zqOXOjQkY2P8SxCkAjzlmg1PpKQ1hzEgob2W18laWD8FfSjNYShqjurevSldtMENyh8DUALgcMeYi4HQKi-"
 />


 Beyond accuracy, we found that asking <code>davinci-003</code> to explain its
 rationale in a zero-shot prompt works reliably, and provides a better
 explanation than a one-shot <code>davinci-002</code> about 75% of the time,
 based on Scale’s human evaluation workforce judging a sample of 100 outputs.

<h3>Text Generation</h3>

 The most notable performance leap comes in how
 <code>davinci-003</code> understands word and sentence structure, especially
 rhyming and syllables. In prompting for a poem that rhymes,
 <code>davinci-003</code> constructs a desired output, whereas
 <code>davinci-002</code> largely fails to implement rhyming.


 <img
 src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F31cb06da7e1a488a91f62ef14feee823"
 />


 <code>davinci-003</code> is also better equipped to handle requests for
 complex output structures, such as iambic pentameter (think Shakespeare).
 <code>davinci-002</code> certainly provides a poem, but ignores the request
 for lines of 5 pairs of stressed and unstressed syllables (buh-BUM). While not
 perfect, <code>davinci-003</code>’s output is significantly more faithful to
 the requested prompt.


 <img
 src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fc3fc8edf1aea432bb7bb5118a6dc8c82"
 />

<h3>Summarization</h3>

 One practical difference between the two variants is verbosity. In our
 experimentation, <code>davinci-003</code> consistently generates longer
 outputs than <code>davinci-002</code> on the same prompts. This is most
 visible in summarization tasks, where the value of the result may depend on
 its brevity.


 In a sample of 60 articles, we tested both variants’ summarization
 capabilities by asking them to summarize the introductory paragraphs of
 lengthy Wikipedia pages. The additional text generated by
 <code>davinci-003</code> is typically beneficial, including information
 central to an article’s importance. However, the resulting summaries are also
 wordy. In our analysis, <code>davinci-003</code> generates outputs about 65%
 longer than <code>davinci-002</code> under identical prompts.


 <img
 src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fbbd0ed222dff4216ab9173d07d176890"
 />


 It will be important for GPT-3 users to bear in mind
 <code>davinci-003</code>’s talkativeness when choosing between the two
 variants. In our testing, given the task of generating a 1-sentence summary,
 <code>davinci-003</code> produced 42 words on average, nearly doubling
 <code>davinci-002</code>‘s average of 23. The shortest summary
 <code>davinci-003</code> gave back was a beefy 32 words, compared to only 12
 words for <code>davinci-002</code>. This tendency extends to multi-sentence
 summaries as well.


 For use cases that value brevity, <code>davinci-002</code> is the safer
 choice. Nevertheless, for many use cases, the longer and more detailed
 completions provided by <code>davinci-003</code> can be beneficial.&nbsp;

<h3>Conclusion</h3>

 Overall, <code>davinci-003</code> is a meaningful improvement from
 <code>davinci-002</code>, especially for zero-shot classification and complex
 text generation.


 While many of the benefits of the latest iteration of GPT-3 are apparent at
 the surface, concrete evaluation metrics ensure models and prompt strategies
 are performing as well as possible.


 Luckily, Spellbook makes that easy, allowing users to run instant programmatic
 evaluation in classification use cases, or sending model outputs to Scale’s
 high-quality workforce to get quantitative evaluation metrics for generative
 use cases.


 <a
 href="https://scale.com/spellbook"
 rel="noopener noreferrer"
 target="_blank"
 >Sign up for Spellbook today!</a
 >


 P.S. If you’re looking for a little dramatic flair in your
 emails to coworkers, <code>davinci-003</code> has you covered. It performs
 significantly better at creatively translating text or even explaining code as
 different celebrities in a zero-shot setting.

 

 BranchCommitmaintestmaintest
 <img
 src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa4f4e03e1cef4d28a44fcf465150aeca"
 />

<pre class="ql-syntax" spellcheck="false">
 Write a short rhyming poem about winter in New York davinci-002
davinci-003
</pre>
In New York City When winter descends
The air is so crisp and chilly
And the wind never ends
The snowflakes are so big and fluffy
You just want to grab a handful
And then make a snowball
People hurry to and fro
Bundled up in coats
Hats, and scarves, and gloves
No one wants to be caught

 Lights are twinkling in the night, It's wintertime in New York City so bright.

The snow is glistening, the wind is cold,
The atmosphere is really bold.
The snowflakes fall with a gentle grace,
Adorning the city with a lovely lace.
The air is crisp, and the people merry,
It's a magical winter in New York City.

Text Generation

 The most notable performance leap comes in how davinci-003 understands word
 and sentence structure, especially rhyming and syllables. In prompting for a
 poem that rhymes, davinci-003 constructs a desired output, whereas davinci-002
 largely fails to implement rhyming.

Content & Language

We evaluate davinci-003 across a range of classification, summarization, and generation tasks. Using Scale Spellbook, the platform for large language model apps, we show where davinci-003 significantly outperforms the prior version and where it still has room to improve.

How Much Better is OpenAI’s Newest GPT-3 Model?

Scale AI is dedicated to helping the Department of Defense (DoD) compete, fight, and win by providing high-quality, government-owned data that can power any AI/ML and autonomy effort.&nbsp;A first step towards this goal is organizing the DoD data layer. Technologies still relatively new for the DoD - like cloud computing and data repositories - are a good start. These are insufficient however, to provide the DoD with data processing capabilities required to fuel large-scale autonomy operations.&nbsp;Scale can provide the next critical layer in the software stack which can bring autonomy efforts out of the labs, into production, and onto the battlefield.
Beyond developing, training, and deploying individual autonomous systems, taking the time to &ldquo;get the data layer right&rdquo; can have broader impact, such as enabling the creation of &ldquo;Data Annexes&rdquo; to go along with Operational Plans (OPLANs).&nbsp;These would allow DoD leaders to provide the US Army&rsquo;s autonomous vehicle fleets with key, pre-validated datasets for critical operations in theater. These would include theater-specific perception models (LiDAR, camera, radar), segmentation-mapped avenues of approach, Order-of-Battle class and type IDs for detection and recognition, EO and SAR imagery mapped to friendly schemes of maneuver, and much more. With a single update, a vehicle fleet would be prepared to move out.&nbsp;
Scale provides the data and machine learning operations (MLOps) pipelines for the world&rsquo;s most ambitious robotics and autonomy teams.&nbsp;As the DoD explores novel approaches to human-machine teaming, it will require a sophisticated, government-owned data strategy with dedicated tooling.&nbsp;This will provide the numerous vehicle platforms and programs with the highest quality data to drive production-grade autonomy. Getting this right is a precursor to mission success.&nbsp;
Scale provides an open data architecture and MLOps pipeline in alignment with the DoD Modular Open Systems Approach (MOSA), decoupled from vehicle ownership.&nbsp;This allows the US Government to reuse training data and algorithms developed within Scale platforms for other programs. Thus the DoD can create a &ldquo;data flywheel&rdquo; that strengthens as more programs develop production-grade autonomy.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fceb82bb90c4741f2a98cdc32b2de19a4">
Scale currently executes billions of <a href="https://scale.com/guides/data-labeling-annotation-guide" target="_blank" rel="noopener noreferrer">annotations</a> per month, maps 3,800 kilometers of roads per week, and leads the industry as the AI perception data provider. Scale partners with self-driving vehicle companies training and building systems ranging from small delivery vehicles to long-haul trucks. Our partners include Kodiak Robotics, Voyage, Optimus Ride, Toyota Research Institute, Embark, General Motors, Honda innovations, Oshkosh, and Nuro - the first autonomous ground delivery vehicle to receive <a href="https://scale.com/customers/nuro" target="_blank" rel="noopener noreferrer">DoT approval</a> for public roads. Additionally, Scale is helping to develop autonomy in other unique domains, including maritime vessels (<a href="https://scale.com/customers/sea-machines-robotics" target="_blank" rel="noopener noreferrer">Sea Machines</a>), autonomous drones (<a href="https://scale.com/customers/skydio" target="_blank" rel="noopener noreferrer">Skydio</a>), and autonomous warehouse systems (<a href="https://scale.com/customers/otto-motors" target="_blank" rel="noopener noreferrer">OTTO Motors</a>).&nbsp;
Through our in-house capabilities, direct partnerships, and an extensive subcontractor network; Scale collects, curates, and <a href="https://scale.com/guides/data-labeling-annotation-guide" target="_blank" rel="noopener noreferrer">processes</a> more ground truth perception data than any other company in the world.
We are a $7B company with real-world applications at high levels of mission-readiness. Scale has over 700 employees focused on cutting edge ML development, human-machine teaming, content understanding, and novel applications for machine learning in the commercial and federal sectors. Our Modular Open System Architecture reduces cost and time to deployment, increases opportunities for iteration, and allows the government to pick and choose the tools and data it needs, all while maintaining full data rights.
Jonathan Hudgins is a member of the Federal Team at Scale where he focuses on bringing Scale&rsquo;s proven commercial solutions to solve Federal Government problems. Jonathan is a former active duty Air Force officer and F-15E Fighter Pilot.

Scale AI is dedicated to helping the Department of Defense (DoD) compete, fight, and win by providing high-quality, government-owned data that can power any AI/ML and autonomy effort.&nbsp; A first step towards this goal is organizing the DoD data layer. Technologies still relatively new for the DoD - like cloud computing and data repositories - are a good start. These are insufficient however, to provide the DoD with data processing capabilities required to fuel large-scale autonomy operations.&nbsp; Scale can provide the next critical layer in the software stack which can bring autonomy efforts out of the labs, into production, and onto the battlefield.Beyond developing, training, and deploying individual autonomous systems, taking the time to “get the data layer right” can have broader impact, such as enabling the creation of “Data Annexes” to go along with Operational Plans (OPLANs).&nbsp; These would allow DoD leaders to provide the US Army’s autonomous vehicle fleets with key, pre-validated datasets for critical operations in theater. These would include theater-specific perception models (LiDAR, camera, radar), segmentation-mapped avenues of approach, Order-of-Battle class and type IDs for detection and recognition, EO and SAR imagery mapped to friendly schemes of maneuver, and much more. With a single update, a vehicle fleet would be prepared to move out.&nbsp;Scale provides the data and machine learning operations (MLOps) pipelines for the world’s most ambitious robotics and autonomy teams.&nbsp; As the DoD explores novel approaches to human-machine teaming, it will require a sophisticated, government-owned data strategy with dedicated tooling.&nbsp; This will provide the numerous vehicle platforms and programs with the highest quality data to drive production-grade autonomy. Getting this right is a precursor to mission success.&nbsp;Scale provides an open data architecture and MLOps pipeline in alignment with the DoD Modular Open Systems Approach (MOSA), decoupled from vehicle ownership.&nbsp; This allows the US Government to reuse training data and algorithms developed within Scale platforms for other programs. Thus the DoD can create a “data flywheel” that strengthens as more programs develop production-grade autonomy.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fceb82bb90c4741f2a98cdc32b2de19a4">Scale currently executes billions of <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a> per month, maps 3,800 kilometers of roads per week, and leads the industry as the AI perception data provider. Scale partners with self-driving vehicle companies training and building systems ranging from small delivery vehicles to long-haul trucks. Our partners include Kodiak Robotics, Voyage, Optimus Ride, Toyota Research Institute, Embark, General Motors, Honda innovations, Oshkosh, and Nuro - the first autonomous ground delivery vehicle to receive <a href="https://scale.com/customers/nuro" rel="noopener noreferrer" target="_blank">DoT approval</a> for public roads. Additionally, Scale is helping to develop autonomy in other unique domains, including maritime vessels (<a href="https://scale.com/customers/sea-machines-robotics" rel="noopener noreferrer" target="_blank">Sea Machines</a>), autonomous drones (<a href="https://scale.com/customers/skydio" rel="noopener noreferrer" target="_blank">Skydio</a>), and autonomous warehouse systems (<a href="https://scale.com/customers/otto-motors" rel="noopener noreferrer" target="_blank">OTTO Motors</a>).&nbsp;Through our in-house capabilities, direct partnerships, and an extensive subcontractor network; Scale collects, curates, and <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">processes</a> more ground truth perception data than any other company in the world.We are a $7B company with real-world applications at high levels of mission-readiness. Scale has over 700 employees focused on cutting edge ML development, human-machine teaming, content understanding, and novel applications for machine learning in the commercial and federal sectors. Our Modular Open System Architecture reduces cost and time to deployment, increases opportunities for iteration, and allows the government to pick and choose the tools and data it needs, all while maintaining full data rights.Jonathan Hudgins is a member of the Federal Team at Scale where he focuses on bringing Scale’s proven commercial solutions to solve Federal Government problems. Jonathan is a former active duty Air Force officer and F-15E Fighter Pilot.

Autonomy Table Stakes:  DoD’s Data Layer

<h3>Background</h3>
The rise of social media platforms has driven the popularity of user-generated content (UGC) and it is changing from the bottom up at a pace faster than ever. Platforms are competing for user attention and engagement - some are winning, but some are falling behind. How can a social media platform stay ahead of the game?
One of the most effective ways to incentivize users to create and engage with content on the platform is by curating and promoting trends. A trend is a type of content repeated and shared amongst creators -&nbsp;a specific subject matter, action, or format that rises in popularity and eventually becomes viral. Common types of trends include dance challenges, transitions, and comedy skits. While videos in a trend follow a similar format, creators usually express their creativity and individuality by adding a personal touch to the videos. For that reason, the variety of the videos within the same trend can be wide and unpredictable.
The key elements of these video trends include the video&rsquo;s image, motion, audio, and text overlay. For example, a transition trend is a trend where the participants show a sudden change in their appearance, often signified by a dramatic sound effect in the background music and sometimes accompanied by dance moves, lip syncing, and text overlay. The lifecycle of a trend can vary from days to months depending on various factors. To remain competitive, social media platforms need to detect the trends while they are still emerging and put them to use immediately by matching them to the right creators and promoting them to the suitable audience.&nbsp;
<a href="https://scale.com/content-understanding" target="_blank" rel="noopener noreferrer">Scale Content Understanding</a> improves platform experiences by enriching content metadata, discovering trend insights, and flagging sensitive content. One feature of Content Understanding is content data enrichment for better trend detection. To facilitate this process, we developed proprietary algorithms to detect trends early, quickly, and thoroughly.&nbsp;
<h3>ML Clustering Overview</h3>
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/449d8647-5509-470c-a6cf-ea38c14a62f3">
We took a human-in-the-loop (HitL) machine learning approach to detect trends. We represent trends as clusters of videos estimated from video-extracted embeddings. Once formed, these clusters are sent to a human pipeline (which we call &ldquo;Cluster Labeling&rdquo;) where taskers answer a series of questions about the videos in a given cluster - namely, if a sample of the videos corresponds to a trend and the potential name of the trend. The following are the phases of the ML Clustering process:
<h3>Phase 1: Embedding Extraction</h3>
To extract embeddings, we use a variety of multi-modal information extraction machine learning models, each of which is trained on large-scale, common datasets with high distribution overlap to the social media domain.
<h3>Phase 2: Clustering</h3>
Once the embeddings are extracted, we perform a clustering pipeline to take in these embeddings and output clusters.
Overview
Given the embeddings, we first perform <a href="https://arxiv.org/pdf/1802.03426.pdf" target="_blank" rel="noopener noreferrer">Uniform Manifold Approximation and Projection (UMAP)</a> dimensionality reduction (inspired by <a href="https://arxiv.org/abs/2203.05794" target="_blank" rel="noopener noreferrer">BERTopic</a>). UMAP takes in high-dimensional data and reduces it down to a lower dimensional space scalably (e.g. UMAP performs computation on tens of thousands of points on the order of tens of seconds). It reduces dimensionality in a way that preserves global Riemannian geometric structure and pairwise distances. On these low-dimensional embeddings, we perform an iterative hierarchical clustering process. Each individual point in embedding space is initially represented as a cluster.&nbsp;We then calculate the proximity between all pairs of clusters. We calculate the proximity of two clusters in a given pair as the variance in embedding space (i.e. the average sum of squares from the joined cluster mean embedding vector) of the points corresponding to if these clusters were to be joined. The two clusters that are closest to each other are joined given the above definition of proximity. This cluster-joining process is performed iteratively until all clusters are above a specified variance that is provided as a hyperparameter.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/24053e18-8fbc-46a8-91ba-4ea274599636">
Stop Condition Variance Hyperparameter
We carefully tune the variance hyperparameter in the stop condition to balance between making the clusters too coarse or too granular. For large variance hyperparameters, the final clusters are allowed to span a wide region in embedding space, meaning they are not semantically similar enough to represent trends (which tend to be extremely specific). Diametrically opposed, for small variance hyperparameters, the clusters end up with too few videos for users in the cluster labeling pipeline to recognize the trend. We therefore carefully tune the stop condition variance hyperparameter. This provides a balance between the clusters being fine enough that they represent groups of videos that are semantically similar enough to correspond to trends, and coarse enough that users from the cluster labeling pipeline can recognize the trend. This hyperparameter is normalized in order to reduce the amount of per-dataset manual tuning.
<h3>Phase 3: Cluster Post-Processing</h3>
To improve the quality and throughput of the taskers, we prune out clusters that are composed of a number of videos that is smaller than our definition of a trend. We also add a post-processing density score to help taskers focus on the clusters which are tightest in embedding space - i.e. the ones that are most likely to be semantically similar. This density score is calculated as the intra-cluster variance of each cluster in the dimensionality-reduced embedding space.
<h3>Conclusion</h3>
Our trend detection product consistently performs at a higher than 90% recall and 90% precision in production. Our HitL ML approach enables us to stay flexible with customization towards specific content niches while maintaining a high throughput. We are able to process tens of millions of videos with a few hours of turnaround. Our priority is to not only detect trends promptly, but also equip our customers with the data and insights on these trends to take actions immediately.&nbsp;
At Scale, we believe investing in data is the key to building better platform experiences and discovering trend insights. If you&rsquo;re interested in learning more about <a href="https://scale.com/content-understanding" target="_blank" rel="noopener noreferrer">Scale Content Understanding</a>, speak to our team today.&nbsp;

<h3>Background</h3>The rise of social media platforms has driven the popularity of user-generated content (UGC) and it is changing from the bottom up at a pace faster than ever. Platforms are competing for user attention and engagement - some are winning, but some are falling behind. How can a social media platform stay ahead of the game?One of the most effective ways to incentivize users to create and engage with content on the platform is by curating and promoting trends. A trend is a type of content repeated and shared amongst creators -&nbsp; a specific subject matter, action, or format that rises in popularity and eventually becomes viral. Common types of trends include dance challenges, transitions, and comedy skits. While videos in a trend follow a similar format, creators usually express their creativity and individuality by adding a personal touch to the videos. For that reason, the variety of the videos within the same trend can be wide and unpredictable.The key elements of these video trends include the video’s image, motion, audio, and text overlay. For example, a transition trend is a trend where the participants show a sudden change in their appearance, often signified by a dramatic sound effect in the background music and sometimes accompanied by dance moves, lip syncing, and text overlay. The lifecycle of a trend can vary from days to months depending on various factors. To remain competitive, social media platforms need to detect the trends while they are still emerging and put them to use immediately by matching them to the right creators and promoting them to the suitable audience.&nbsp;<a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">Scale Content Understanding</a> improves platform experiences by enriching content metadata, discovering trend insights, and flagging sensitive content. One feature of Content Understanding is content data enrichment for better trend detection. To facilitate this process, we developed proprietary algorithms to detect trends early, quickly, and thoroughly.&nbsp;<h3>ML Clustering Overview</h3><img src="https://lh5.googleusercontent.com/pyuhI7_2AOewmVrl9oN1O1384a6ETUTQcMBnrRULqomQLo79uFFK_nUBi56VyV86vbcCWroj6htZw38n-xGNNNjuTWmO3l5ovBVbH3JO90n_h4wnlgwzHCtFPbG3q0oaECsQAPHB1fLNT2v8L53SokbiWcFQHYGQgXC1ixo_rW12-zkpx2h7ZC88NhD-nw">We took a human-in-the-loop (HitL) machine learning approach to detect trends. We represent trends as clusters of videos estimated from video-extracted embeddings. Once formed, these clusters are sent to a human pipeline (which we call “Cluster Labeling”) where taskers answer a series of questions about the videos in a given cluster - namely, if a sample of the videos corresponds to a trend and the potential name of the trend. The following are the phases of the ML Clustering process:<h3>Phase 1: Embedding Extraction</h3>To extract embeddings, we use a variety of multi-modal information extraction machine learning models, each of which is trained on large-scale, common datasets with high distribution overlap to the social media domain.<h3>Phase 2: Clustering</h3>Once the embeddings are extracted, we perform a clustering pipeline to take in these embeddings and output clusters.OverviewGiven the embeddings, we first perform <a href="https://arxiv.org/pdf/1802.03426.pdf" rel="noopener noreferrer" target="_blank">Uniform Manifold Approximation and Projection (UMAP)</a> dimensionality reduction (inspired by <a href="https://arxiv.org/abs/2203.05794" rel="noopener noreferrer" target="_blank">BERTopic</a>). UMAP takes in high-dimensional data and reduces it down to a lower dimensional space scalably (e.g. UMAP performs computation on tens of thousands of points on the order of tens of seconds). It reduces dimensionality in a way that preserves global Riemannian geometric structure and pairwise distances. On these low-dimensional embeddings, we perform an iterative hierarchical clustering process. Each individual point in embedding space is initially represented as a cluster.&nbsp; We then calculate the proximity between all pairs of clusters. We calculate the proximity of two clusters in a given pair as the variance in embedding space (i.e. the average sum of squares from the joined cluster mean embedding vector) of the points corresponding to if these clusters were to be joined. The two clusters that are closest to each other are joined given the above definition of proximity. This cluster-joining process is performed iteratively until all clusters are above a specified variance that is provided as a hyperparameter.<img src="https://lh4.googleusercontent.com/keSURwh73ya9IjUsA_pZWm8z4rhGY6YZiaEYn_885VoFWbCev4cKNksO8eKu5Kgom4ncJtsU4Y-DZVQtaA-fx4jylidiqVUcCczzyEvdYjO486XjNN5OY5SWRdU2Lj91rL4V_xBdA4to8O-UDi9lwE2Yu-bVduGg_-5VHuRu0JFKT-nWjnI5OTDQApHrcQ">Stop Condition Variance HyperparameterWe carefully tune the variance hyperparameter in the stop condition to balance between making the clusters too coarse or too granular. For large variance hyperparameters, the final clusters are allowed to span a wide region in embedding space, meaning they are not semantically similar enough to represent trends (which tend to be extremely specific). Diametrically opposed, for small variance hyperparameters, the clusters end up with too few videos for users in the cluster labeling pipeline to recognize the trend. We therefore carefully tune the stop condition variance hyperparameter. This provides a balance between the clusters being fine enough that they represent groups of videos that are semantically similar enough to correspond to trends, and coarse enough that users from the cluster labeling pipeline can recognize the trend. This hyperparameter is normalized in order to reduce the amount of per-dataset manual tuning.<h3>Phase 3: Cluster Post-Processing</h3>To improve the quality and throughput of the taskers, we prune out clusters that are composed of a number of videos that is smaller than our definition of a trend. We also add a post-processing density score to help taskers focus on the clusters which are tightest in embedding space - i.e. the ones that are most likely to be semantically similar. This density score is calculated as the intra-cluster variance of each cluster in the dimensionality-reduced embedding space.<h3>Conclusion</h3>Our trend detection product consistently performs at a higher than 90% recall and 90% precision in production. Our HitL ML approach enables us to stay flexible with customization towards specific content niches while maintaining a high throughput. We are able to process tens of millions of videos with a few hours of turnaround. Our priority is to not only detect trends promptly, but also equip our customers with the data and insights on these trends to take actions immediately. &nbsp;At Scale, we believe investing in data is the key to building better platform experiences and discovering trend insights. If you’re interested in learning more about <a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">Scale Content Understanding</a>, speak to our team today.&nbsp;

Discovering short-form video trends is challenging. In this blog post, we explain how we use an ML + HiTL approach for faster and higher-quality trend detection.

Discovering Emerging Short Form Video Trends with Clustering and HitL

Becoming a veteran requires a transition from the regimented lifestyle as a service member into the civilian world. The majority of veterans <a href="https://www.cfr.org/backgrounder/demographics-us-military" target="_blank" rel="noopener noreferrer">joined out of high school or college</a>, and their adult lives have been shaped by their military service. We entrust our service members at a young age with responsibilities that they otherwise would never encounter in the civilian world. These jobs not only require a highly trained technical force, but are often life or death situations.
In my case, I was only 20 when I received orders to my ship. I was responsible for operating &amp; maintaining the computers and networks that powered the multi-million dollar AEGIS Combat system. By the time I was 22, I was on a 7 month deployment in the Persian Gulf, leading a team of fellow technicians supporting combat operations in Iraq &amp; Afghanistan during the 2011 withdrawal from Iraq.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa540352ee602468a940cb12fc334d5e6">
My experience isn&rsquo;t unique. The average age of the US Military service member is<a href="https://download.militaryonesource.mil/12038/MOS/Reports/2018-demographics-report.pdf" target="_blank" rel="noopener noreferrer"> 28 years old</a>, with the vast majority of service members serving one tour and becoming a veteran before the age of 25. My fellow U.S. Navy Veteran at Scale, Nick Falconi, was only 25 when the Navy trusted him to oversee the operations of the nuclear reactor that powered the submarine he served onboard. He did this alone while in port as the Duty Officer, and acted as Senior Officer while the other Senior Officers slept at sea.
Serving in the military requires a commitment that is hard to describe. Having the privilege to do amazing things at such a young age makes it hard to define an identity outside of the military. Becoming a veteran means giving up the military which has defined your adult life up until that point. One day you are a fighter pilot, special forces, or intelligence analyst in the military, and the next you are just a person trying to figure out how to succeed in the civilian world.
It&rsquo;s already hard to give that up, and enter the civilian world, but it&rsquo;s even harder to translate that experience into a civilian career. After I left the Navy and transitioned, I made the decision to go to college back home in St. Louis, but I struggled being a 23 year old freshman in classes with 18 year old kids. After a year I decided to enter the workforce while still pursuing my degree. I had difficulty finding entry level IT positions, even though the AEGIS weapon system I supported was similar to the computers &amp; network equipment found at most companies. I use the same technical skills I did in the Navy daily in my current role as the Lead of the Infrastructure &amp; Security team for Scale&rsquo;s Federal Team.
It wasn&rsquo;t until I started getting involved in the St. Louis Veterans Community and volunteering that I found the mentors and resources that helped me succeed. I learned from other veterans in the St. Louis community like Steve Wahle &amp; Micheal Bamber how to use my military experience to my advantage. They introduced me to community leaders like Maxine Clark, the founder and former CEO of Build-A-Bear Workshops, who helped me land my first civilian job supporting the computers and networks that powered the Build-A-Bear retail empire.&nbsp;
I strongly believe a successful transition to civilian life is not only key to reintegrating veterans into the community, but it is also a good business decision. Veterans have deep technical knowledge &amp; leadership skills, along with a tenacity that is hard to train that makes successful employees who can jump right into tough problems. Scale recognizes this, and has been successful in its commitment to hire veterans. I&rsquo;m proud to say that Scale&rsquo;s veteran population is nearly 5% of the US based workforce exceeding the <a href="https://www.prweb.com/releases/new_research_from_findem_sheds_light_on_status_of_veterans_in_the_workforce/prweb17533903.htm" target="_blank" rel="noopener noreferrer">percentage of veterans in the tech workforce,</a> and one-third of veteran employees are recently separated.&nbsp;
I encourage everyone this Veteran&rsquo;s Day to commit to helping new veterans. If you are a transitioning service member looking for a new challenge &amp; mission, consider <a href="https://scale.com/blog" target="_blank" rel="noopener noreferrer">Scale.</a> Our mission is to accelerate the development of AI. If that sounds exciting to you, let's talk.

Becoming a veteran requires a transition from the regimented lifestyle as a service member into the civilian world. The majority of veterans <a href="https://www.cfr.org/backgrounder/demographics-us-military" rel="noopener noreferrer" target="_blank">joined out of high school or college</a>, and their adult lives have been shaped by their military service. We entrust our service members at a young age with responsibilities that they otherwise would never encounter in the civilian world. These jobs not only require a highly trained technical force, but are often life or death situations.In my case, I was only 20 when I received orders to my ship. I was responsible for operating &amp; maintaining the computers and networks that powered the multi-million dollar AEGIS Combat system. By the time I was 22, I was on a 7 month deployment in the Persian Gulf, leading a team of fellow technicians supporting combat operations in Iraq &amp; Afghanistan during the 2011 withdrawal from Iraq.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa540352ee602468a940cb12fc334d5e6">My experience isn’t unique. The average age of the US Military service member is<a href="https://download.militaryonesource.mil/12038/MOS/Reports/2018-demographics-report.pdf" rel="noopener noreferrer" target="_blank"> 28 years old</a>, with the vast majority of service members serving one tour and becoming a veteran before the age of 25. My fellow U.S. Navy Veteran at Scale, Nick Falconi, was only 25 when the Navy trusted him to oversee the operations of the nuclear reactor that powered the submarine he served onboard. He did this alone while in port as the Duty Officer, and acted as Senior Officer while the other Senior Officers slept at sea.Serving in the military requires a commitment that is hard to describe. Having the privilege to do amazing things at such a young age makes it hard to define an identity outside of the military. Becoming a veteran means giving up the military which has defined your adult life up until that point. One day you are a fighter pilot, special forces, or intelligence analyst in the military, and the next you are just a person trying to figure out how to succeed in the civilian world.It’s already hard to give that up, and enter the civilian world, but it’s even harder to translate that experience into a civilian career. After I left the Navy and transitioned, I made the decision to go to college back home in St. Louis, but I struggled being a 23 year old freshman in classes with 18 year old kids. After a year I decided to enter the workforce while still pursuing my degree. I had difficulty finding entry level IT positions, even though the AEGIS weapon system I supported was similar to the computers &amp; network equipment found at most companies. I use the same technical skills I did in the Navy daily in my current role as the Lead of the Infrastructure &amp; Security team for Scale’s Federal Team.It wasn’t until I started getting involved in the St. Louis Veterans Community and volunteering that I found the mentors and resources that helped me succeed. I learned from other veterans in the St. Louis community like Steve Wahle &amp; Micheal Bamber how to use my military experience to my advantage. They introduced me to community leaders like Maxine Clark, the founder and former CEO of Build-A-Bear Workshops, who helped me land my first civilian job supporting the computers and networks that powered the Build-A-Bear retail empire.&nbsp;I strongly believe a successful transition to civilian life is not only key to reintegrating veterans into the community, but it is also a good business decision. Veterans have deep technical knowledge &amp; leadership skills, along with a tenacity that is hard to train that makes successful employees who can jump right into tough problems. Scale recognizes this, and has been successful in its commitment to hire veterans. I’m proud to say that Scale’s veteran population is nearly 5% of the US based workforce exceeding the <a href="https://www.prweb.com/releases/new_research_from_findem_sheds_light_on_status_of_veterans_in_the_workforce/prweb17533903.htm" rel="noopener noreferrer" target="_blank">percentage of veterans in the tech workforce,</a> and one-third of veteran employees are recently separated.&nbsp;I encourage everyone this Veteran’s Day to commit to helping new veterans. If you are a transitioning service member looking for a new challenge &amp; mission, consider <a href="https://scale.com/blog" rel="noopener noreferrer" target="_blank">Scale.</a> Our mission is to accelerate the development of AI. If that sounds exciting to you, let's talk.

 If you are a transitioning service member looking for a new challenge & mission, consider Scale.

Scale is committed to hiring veterans

Veterans Day @ Scale

The 2022 TransformX conference may now be behind us, but you can still catch all of our sessions on-demand! Nearly 32,000 virtual and in-person attendees joined the 3-day event, which included almost 80 sessions and 108 speakers. 

TransformX 2022 Highlights and Recaps

Today, we&rsquo;re excited to announce the Early Access of Scale Forge (Beta) for AI-generated product imagery in seconds. With just a few clicks, marketers and brands can create stunning visual content for ad creatives, campaigns, and social media. Learn more and <a href="https://scale.com/catalog" target="_blank" rel="noopener noreferrer">sign-up for Early Access</a>.
<h2>Unlocking Stunning Product Imagery</h2>
Currently, retailers and advertisers are limited by the amount and quality of product photography they have to create an engaging shopping experience for customers. The investment needed for photoshoots, scale of product catalogs, and the diverse tastes of audiences adds to this burden. This limits the growth of businesses, and ultimately their ability to communicate their story and unique product offering to their audience.
Building on top of our existing strengths and technology in eCommerce, we built Forge to solve this problem.
<h2>Catalog Forge</h2>
With Forge, we&rsquo;re enabling customers to create AI-generated imagery via an easy-to-use design studio. Forge uses novel ML methods to allow for high-fidelity product and brand preservation for retail products.&nbsp;
With Forge, teams can:
<ol>
<li>Experiment and prototype quickly to decide the aesthetic of images</li>
<li>Operationalize and scale product image generation to enable servicing thousands of brands and products</li>
<li>Improve conversion with enhanced product images to maximize ROI&nbsp;&nbsp;</li>
</ol>
<a href="https://scaleai.wistia.com/medias/sk212njbxf" target="_blank" rel="noopener noreferrer">Video</a>
To get started, simply provide your product image, select a scene or describe your own, and create imagery through two ways:&nbsp;
<ol>
<li>Scene Enrichment: Create unique customizable scenes to generate product images that align with your brand values and product identity. Retain the original pose of each image. Best for: rigid objects</li>
<li>Full Transformation: Experiment with different poses, zoom levels, whole scenes, complementary products, and more. Only available for select demo products right now. Best for: clothing, fashion accessories</li>
</ol>
Forge enables you to create unique and memorable product images for your brand powered by AI.
<h2>New Era of Creativity in eCommerce and Retail&nbsp;</h2>
We&rsquo;re thrilled to introduce Scale Forge for eCommerce. We&rsquo;re excited to see what companies can create once they have access to the best quality AI tools. At Scale, we believe investing in data is the key to unlocking success for eCommerce companies, and we&rsquo;re just getting started.&nbsp;
To get Early Access for Forge, <a href="https://scale.com/catalog" target="_blank" rel="noopener noreferrer">sign-up for the waitlist</a>.

Today, we’re excited to announce the Early Access of Scale Forge (Beta) for AI-generated product imagery in seconds. With just a few clicks, marketers and brands can create stunning visual content for ad creatives, campaigns, and social media. Learn more and <a href="https://scale.com/catalog" rel="noopener noreferrer" target="_blank">sign-up for Early Access</a>.<h2>Unlocking Stunning Product Imagery </h2>Currently, retailers and advertisers are limited by the amount and quality of product photography they have to create an engaging shopping experience for customers. The investment needed for photoshoots, scale of product catalogs, and the diverse tastes of audiences adds to this burden. This limits the growth of businesses, and ultimately their ability to communicate their story and unique product offering to their audience.Building on top of our existing strengths and technology in eCommerce, we built Forge to solve this problem.<h2>Catalog Forge</h2>With Forge, we’re enabling customers to create AI-generated imagery via an easy-to-use design studio. Forge uses novel ML methods to allow for high-fidelity product and brand preservation for retail products.&nbsp;With Forge, teams can:<ol><li>Experiment and prototype quickly to decide the aesthetic of images</li><li>Operationalize and scale product image generation to enable servicing thousands of brands and products</li><li>Improve conversion with enhanced product images to maximize ROI&nbsp;&nbsp;</li></ol><a href="https://scaleai.wistia.com/medias/sk212njbxf" rel="noopener noreferrer" target="_blank">Video</a>To get started, simply provide your product image, select a scene or describe your own, and create imagery through two ways:&nbsp;<ol><li>Scene Enrichment: Create unique customizable scenes to generate product images that align with your brand values and product identity. Retain the original pose of each image. Best for: rigid objects</li><li>Full Transformation: Experiment with different poses, zoom levels, whole scenes, complementary products, and more. Only available for select demo products right now. Best for: clothing, fashion accessories</li></ol>Forge enables you to create unique and memorable product images for your brand powered by AI.<h2>New Era of Creativity in eCommerce and Retail&nbsp;</h2>We’re thrilled to introduce Scale Forge for eCommerce. We’re excited to see what companies can create once they have access to the best quality AI tools. At Scale, we believe investing in data is the key to unlocking success for eCommerce companies, and we’re just getting started.&nbsp;To get Early Access for Forge, <a href="https://scale.com/catalog" rel="noopener noreferrer" target="_blank">sign-up for the waitlist</a>.

Retail & eCommerce

Catalog Forge is the fastest way to create AI-generated product imagery in seconds. Forge is now in Early Access.

Announcing Scale Forge: Available for Early Access 

Today we kicked off our third annual conference, TransformX, where the world's brightest AI leaders, visionaries, practitioners, and researchers engaged in conversations around the present and future of AI/ML.
At this year&rsquo;s event, we felt it was important to shine a light on how AI is going to define the next era of technology and power. Just like at the dawn of the internet, or cloud computing before it, the businesses that leverage AI will have an incredible competitive advantage to dominate their industries for years to come.&nbsp;
In fact, for many businesses, we&rsquo;re already at a nexus where it&rsquo;s AI or die. And that&rsquo;s because not all companies are equipped with the same resources to leverage the power of AI. Some organizations, often the biggest tech companies, are AI builders&mdash;they have the talent, resources, and experience to build their own, differentiated AI. But they often lack the high-quality data needed to feed their algorithms. On the other hand, there are organizations that aren&rsquo;t rooted in technology, but are prepared to leverage AI as a key asset and differentiator. They are looking to leverage this technology into their businesses without having to build their own ML team.
Whether an organization is already building AI or is at the starting line wanting to apply AI, it&rsquo;s clear that all organizations can reap tangible benefits from AI if they have the right tools and solutions tailored to their business. That&rsquo;s why today at TransformX, Scale introduced a suite of new products to unlock and operationalize AI for everyone &ndash; from startups to researchers and Fortune 500 companies to the U.S. government. These tools have already been battle tested in early access beta by companies such as <a href="https://opensea.io/" target="_blank" rel="noopener noreferrer">OpenSea</a>, <a href="https://www.voxelai.com/" target="_blank" rel="noopener noreferrer">Voxel</a> and <a href="https://picklerobot.com/" target="_blank" rel="noopener noreferrer">Pickle Robot</a>, and now they are fully available for everyone to adapt, develop and scale their use of AI.&nbsp;
<h2>Studio</h2>
Some labeling projects require in-depth subject-matter expertise. For example, medical initiatives have unique terminology and require deep domain knowledge. Studio provides organizations with the capability to run high quality, efficient labeling projects with their own team of experts. Features include the ability to pre-label datasets to reduce labeling time and auto-annotation to make it easy for anyone to label complex objects by simply drawing a box around the specific object. Studio supports all data types, including images, videos, audio, and text. Any team can quickly get started with Studio by signing up <a href="https://scale.com/studio" target="_blank" rel="noopener noreferrer">here</a>.&nbsp;
<h2>Nucleus Models&nbsp;</h2>
As many ML teams know, shipping a model isn&rsquo;t the end of the AI journey. To ensure a model performs well in all scenarios and avoids model failure in production, it is critical to test key scenarios beforehand. In 2020, we launched Nucleus, a new way to develop and test ML models. We are now improving Nucleus with new ML model testing functionalities that help catch regression over time and help teams to choose the best model through efficient model comparisons and scenario-level insights. Learn more about Nucleus and its model features <a href="https://scale.com/nucleus" target="_blank" rel="noopener noreferrer">here</a>.
<h2>Content Understanding AI</h2>
One of the biggest challenges for companies that invest in user-generated content is to recommend relevant content that keeps users engaged. With Content Understanding AI, product teams can easily enrich user-generated content to customize and iterate their content&rsquo;s taxonomy - ultimately increasing engagement. Data enrichment also gives product teams insights into their user&rsquo;s content, such as sentiment and mood, to help identify growth areas. Additionally, trend detection leverages ML techniques like clustering, topic aggregation and extraction, as well as Scale human-in-the-loop features to detect and understand new content trends surfacing on platforms. Learn more about Scale&rsquo;s Content Understanding <a href="https://scale.com/content-understanding" target="_blank" rel="noopener noreferrer">here</a>.&nbsp;
<h2>Document AI Go&nbsp;</h2>
In 2020, we launched Document AI to process complex documents up to near perfect accuracy in seconds. This year, we launched Document AI Go. Document AI Go is a self-serve machine learning platform that enables companies to quickly classify and extract data from their documents in just a matter of seconds. Unlike competing products, Document AI and Document AI Go are powered by foundation models and don&rsquo;t rely on templates, so they can handle the most complex and unstructured documents. Any organization can quickly sign up and start uploading documents <a href="https://scale.com/document-ai#pricing" target="_blank" rel="noopener noreferrer">here</a>.
We&rsquo;re thrilled to introduce these tools and put them in the hands of every company. We can only imagine what&rsquo;s possible when every company can access the best quality AI tools and tailored models. I believe that by making it easier for more companies to unlock and operationalize AI, we will experience a world where AI meaningfully supports humans as they solve the world&rsquo;s biggest problems.

Today we kicked off our third annual conference, TransformX, where the world's brightest AI leaders, visionaries, practitioners, and researchers engaged in conversations around the present and future of AI/ML.At this year’s event, we felt it was important to shine a light on how AI is going to define the next era of technology and power. Just like at the dawn of the internet, or cloud computing before it, the businesses that leverage AI will have an incredible competitive advantage to dominate their industries for years to come.&nbsp;In fact, for many businesses, we’re already at a nexus where it’s AI or die. And that’s because not all companies are equipped with the same resources to leverage the power of AI. Some organizations, often the biggest tech companies, are AI builders—they have the talent, resources, and experience to build their own, differentiated AI. But they often lack the high-quality data needed to feed their algorithms. On the other hand, there are organizations that aren’t rooted in technology, but are prepared to leverage AI as a key asset and differentiator. They are looking to leverage this technology into their businesses without having to build their own ML team.Whether an organization is already building AI or is at the starting line wanting to apply AI, it’s clear that all organizations can reap tangible benefits from AI if they have the right tools and solutions tailored to their business. That’s why today at TransformX, Scale introduced a suite of new products to unlock and operationalize AI for everyone – from startups to researchers and Fortune 500 companies to the U.S. government. These tools have already been battle tested in early access beta by companies such as <a href="https://opensea.io/" rel="noopener noreferrer" target="_blank">OpenSea</a>, <a href="https://www.voxelai.com/" rel="noopener noreferrer" target="_blank">Voxel</a> and <a href="https://picklerobot.com/" rel="noopener noreferrer" target="_blank">Pickle Robot</a>, and now they are fully available for everyone to adapt, develop and scale their use of AI.&nbsp;<h2>Studio</h2>Some labeling projects require in-depth subject-matter expertise. For example, medical initiatives have unique terminology and require deep domain knowledge. Studio provides organizations with the capability to run high quality, efficient labeling projects with their own team of experts. Features include the ability to pre-label datasets to reduce labeling time and auto-annotation to make it easy for anyone to label complex objects by simply drawing a box around the specific object. Studio supports all data types, including images, videos, audio, and text. Any team can quickly get started with Studio by signing up <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">here</a>.&nbsp;<h2>Nucleus Models&nbsp;</h2>As many ML teams know, shipping a model isn’t the end of the AI journey. To ensure a model performs well in all scenarios and avoids model failure in production, it is critical to test key scenarios beforehand. In 2020, we launched Nucleus, a new way to develop and test ML models. We are now improving Nucleus with new ML model testing functionalities that help catch regression over time and help teams to choose the best model through efficient model comparisons and scenario-level insights. Learn more about Nucleus and its model features <a href="https://scale.com/nucleus" rel="noopener noreferrer" target="_blank">here</a>.<h2>Content Understanding AI</h2>One of the biggest challenges for companies that invest in user-generated content is to recommend relevant content that keeps users engaged. With Content Understanding AI, product teams can easily enrich user-generated content to customize and iterate their content’s taxonomy - ultimately increasing engagement. Data enrichment also gives product teams insights into their user’s content, such as sentiment and mood, to help identify growth areas. Additionally, trend detection leverages ML techniques like clustering, topic aggregation and extraction, as well as Scale human-in-the-loop features to detect and understand new content trends surfacing on platforms. Learn more about Scale’s Content Understanding <a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">here</a>.&nbsp;<h2>Document AI Go&nbsp;</h2>In 2020, we launched Document AI to process complex documents up to near perfect accuracy in seconds. This year, we launched Document AI Go. Document AI Go is a self-serve machine learning platform that enables companies to quickly classify and extract data from their documents in just a matter of seconds. Unlike competing products, Document AI and Document AI Go are powered by foundation models and don’t rely on templates, so they can handle the most complex and unstructured documents. Any organization can quickly sign up and start uploading documents <a href="https://scale.com/document-ai#pricing" rel="noopener noreferrer" target="_blank">here</a>.We’re thrilled to introduce these tools and put them in the hands of every company. We can only imagine what’s possible when every company can access the best quality AI tools and tailored models. I believe that by making it easier for more companies to unlock and operationalize AI, we will experience a world where AI meaningfully supports humans as they solve the world’s biggest problems.

Government

Scale Rapid

At TransformX this year, we're sharing details on Studio, Nucleus Models, Content Understanding AI, and Document AI Go.

TransformX 2022: Operationalizing AI

Last week, and every October, we celebrated National Coming Out Day. The celebration started in 1988 on the first anniversary of the second National March on Washington for Lesbian and Gay Rights in October 1987.&nbsp;
<a href="https://lgbtq.gmu.edu/programs-services/national-coming-out-day/" target="_blank" rel="noopener noreferrer">The march</a>'s goal was to draw attention to the federal government&rsquo;s inaction in confronting the AIDS crisis and the Supreme Court&rsquo;s 1986 ruling upholding Georgia&rsquo;s anti-gay law. According to NCOD (National Coming Out Day), protesters flooded D.C. over a five day period and demanded legal recognition for gay and lesbian couples and the funding for AIDS research. The march also marked<a href="https://19thnews.org/2021/10/the-history-of-national-coming-out-day-contains-both-pride-and-pain/" target="_blank" rel="noopener noreferrer"> the unveiling</a> of the AIDS memorial quilt, a massive patchwork honoring those lost to the virus, and at the time an unprecedented show of support for gay rights.
National Coming Out Day is meant to be a celebration of individuals within the LGBTQA+ community who openly identify as lesbian, gay, bisexual, transgender, non-binary, queer, asexual or intersex. It&rsquo;s a day for honoring the act of coming out, and all the hopes, fears, dreams and expectations for the future.
Whether it is at home, work, or in your community, coming out is the first step to living authentically. Scale is committed to creating a culture of inclusion that supports all Scaliens &ndash; no matter how they identify.
We want to express gratitude to the Scaliens who have opened up to share their stories for this year&rsquo;s National Coming Out Day:
<h2>Abe</h2>
I grew up in a town of 1300 people in the middle of Southern Illinois. Where the pressures of never coming out and the expectations of settling down were standard.
We moved from Missouri to Illinois around that time to help my grandma after my grandpa had passed suddenly of cancer.&nbsp;She lived on a farm in the middle of nowhere and that&rsquo;s where me and my twin brother, along with an older brother and sister stayed for a few years until we moved to another house in the country when I was 9. During my tenure in grade school, there were a lot of slurs thrown around by kids who didn&rsquo;t really understand what they meant.&nbsp;I never used them personally, because that's not who I&rsquo;ve ever been. But I know if one of those slurs was directed at someone, it was a really bad thing and meant that kid was now viewed as less than a human.&nbsp;So I tried to fly under the radar.&nbsp;
In middle school, I tried dating girls, because that was what you were supposed to do: date women, get married and have kids. I even pined after my best friend since the 4th grade until we were out of high school, but she never really returned the way I felt.&nbsp;
When I got to high school, a kid in the grade above me was the first openly gay person I&rsquo;d ever met.&nbsp;I admired the heck out of him for doing something I thought I could never do, and also didn&rsquo;t understand why he&rsquo;d want to subject himself to the small-minded mentality of the town we had grown up in.&nbsp;&nbsp;
I finally graduated high school in 2000 and went to a small community college over by Centralia, IL.&nbsp;While going to college there, I was working at a gas station and an attractive male customer came in&hellip;that&rsquo;s when I had my &ldquo;holy CRAP, I&rsquo;m gay&rdquo; moment. I think many of you know what I&rsquo;m talking about: the time you finally admit to yourself that you&rsquo;re definitely not straight.&nbsp;
I met the person who would become my future wife on the first day of freshman orientation.&nbsp;We were inseparable. We did everything together, enjoyed all the same activities like driving around, singing songs on the radio, and just enjoyed being around each other.&nbsp;After a few years, I thought to myself, &ldquo;If there was anyone you could make a straight life with, it would be this woman.&rdquo; So, I did what anybody who came from a town of 1,300 people would do: I asked her to marry me, driving down IL-4 one night.&nbsp;I was between Stelleville and Sparta when I popped the question&hellip;while driving&hellip;and didn&rsquo;t have a ring. We were married on November 6th, 2004 at what was supposed to be a beautiful ceremony outside her parents house.
Even before I had admitted to myself who I truly was, I had explored being with men.&nbsp;With that being said, a few months after I was married, I met another known gay person from my hometown (who had graduated high school right before I had entered) and struck up a friendship with him.&nbsp;Problem was, I ended up kissing him after the friendship started.&nbsp;Now this was the first time I had ever kissed another man, and my whole life changed.&nbsp;I couldn&rsquo;t eat and seriously lost 40 pounds in 2 weeks.&nbsp;I couldn&rsquo;t sleep, because I just thought I was always going to be married to my wife and now that I had kissed another man, I knew I couldn&rsquo;t go on living as a straight man.
So one night, I came home after a late shift, woke my wife up and told her the news, &ldquo;Honey, I don&rsquo;t know how to tell you this, but I&rsquo;m gay.&nbsp;Here&rsquo;s how it happened [see story above] and I&rsquo;m truly sorry.&nbsp;I thought I could be straight and I just can&rsquo;t be anymore.&rdquo;&nbsp;There was a lot of crying that night between the 2 of us, a lot of discussion about the future and what it looked like for us, and how to start tomorrow off. I ended up coming out to everyone else that was important around me for the next few weeks and really received love and support from 99% of my family unit (I&rsquo;ve always had the philosophy that family is the ones you choose as your family, so that doesn&rsquo;t just mean people I&rsquo;m related to).&nbsp;We ended up selling the house and 1 year and 1 month after we were married we officially signed the divorce papers.&nbsp;
I still look back on that night with awe on how I ever had the courage to do it.&nbsp;I know I hurt some people, especially my ex-wife, but we still talk to this day and joke about that time in our life. She&rsquo;s remarried with a beautiful daughter and I&rsquo;ve even met her new husband.&nbsp;I&rsquo;ve since moved to St. Louis where I try to be a beacon in the community, including heading volunteer efforts for an LGBTQ+ group I&rsquo;m a member of, recently chairing a committee that helped raise $1500 for The Metro Trans Umbrella Group (MTUG) in one night.&nbsp;
Yes, it was hard to come out, but I never liked closets anyway.&nbsp;The sunshine is so much better outside of them.
<h2>Willow</h2>
I knew from an early age that I was gay and transgender, but it took me many years to come out. The social stigma felt very high. Gayness was used as an insult, and transgender people were a bad punchline at best.
It took me 14 years of growing up to decide that I didn&rsquo;t really care about the social stigma. During that time, I had focused on becoming an engineer and a soldier, stereotypically male professions. While deployed in Afghanistan, I was struggling to understand why I felt more depressed about the thought of returning home than staying in theater, and I realized that I was living my life inauthentically in a way that was leaving me devastatingly unhappy.
When I got back home, I came out as queer and made plans to leave the military. In 2014, transgender people were still barred for military service, and while it seemed like that might change, I figured it would be a messy policy process with too much back and forth for me to want to live through (unfortunately this turned out to be correct).&nbsp;The first person I came out to as transgender was my long term girlfriend, who was incredibly supportive and caring. Next, I came out to my parents who said they loved and supported me and promised to assist in any way they could with transition. My immediate supervisor was also supportive, which made managing my exit from the military much easier and simpler. Even the most supportive coming out was very hard for me psychologically: it felt like a terrible admission of fragility and stigma. How much more difficult for those without support?
I left the military, went back to school, and changed careers and genders. It was very challenging emotionally, but bluntly far easier for me than most other transgender people as a result of relative socioeconomic privilege and education, and due to the supportive friends and family around me. For this reason, it has always been important to me to be out as queer and transgender in my social life and business life. I do not treat these aspects of who I am as secrets, and I hope that being out makes coming out less of a burden for other people over time.
<h2>Richard&nbsp;</h2>
I knew I was gay years before I met an openly gay person. I was 11 years old and watching TV when all of a sudden I found myself attracted to a male actor. For whatever reason, in that moment, I had no doubt that I was gay. I didn&rsquo;t want to be - &ldquo;gay&rdquo; was a prejorative even among 5th graders in midwest suburbia - but it was a fundamental truth, as undeniable as 2+2=4.
It never occurred to me that I could hide this forever. Living in the closet requires telling dozens of lies daily. Some are overt. Many are subtle, a slight blur of the truth. It&rsquo;s an exhausting existence, one that I personally was not capable of. Rationally, I knew that I had nothing to be ashamed of. But as much as that made logical sense, it was still emotionally terrifying - would I lose my friends? Could I have kids and build a family? Could I have a &ldquo;respectable&rdquo; career and be a &ldquo;respectable&rdquo; member of society?&nbsp;
I&rsquo;m a big believer in &ldquo;getting your reps in&rdquo; - you need to practice in order to build muscles, whether they&rsquo;re physical or emotional. Coming out isn&rsquo;t a one-time thing. You come out over and over again. To your friends&rsquo; parents. To new colleagues. To old friends you didn&rsquo;t really keep in touch with, and you aren&rsquo;t quite sure if the gossip reached them. It took me a few years to learn how to come out without showing my fear externally, and a few more to not feel it internally either. It took dozens of reps to rewire the part of my brain that would go into overdrive - that flight or fight response that kicks in when you subconsciously perceive that you&rsquo;re in danger.
As a teenager, coming out was the hardest thing I had ever done in my life. Arguably, it still is. There were no openly gay students at my high school. There were no openly gay men in my college fraternity. It was terrifying, but what was once a source of terror is now a source of pride.
One of my fondest memories was visiting my brother in 2012. I was 20 years old, and he had joined some small startup called Dropbox in some far away place called San Francisco. I was eating lunch served by company chefs in the office cafeteria - that in itself was a novel concept - and overheard a same-sex couple, surrounded by friends and colleagues, talking about their baby. My mind was blown by how at ease they were, and how &ldquo;normal&rdquo; it all felt. At that moment, I really felt that I could both be gay and have the foundational things I wanted - friends, family, and a career. My dream was a possibility, and it was up to me to make it a reality. For this, I will always be grateful.
During my lifetime, the world has overall become a more accepting place. I&rsquo;m optimistic that, even if there are some bumps in the road, we&rsquo;ll continue to make progress, and more and more people will have the freedom to live authentically.&nbsp;

Last week, and every October, we celebrated National Coming Out Day. The celebration started in 1988 on the first anniversary of the second National March on Washington for Lesbian and Gay Rights in October 1987.&nbsp;<a href="https://lgbtq.gmu.edu/programs-services/national-coming-out-day/" rel="noopener noreferrer" target="_blank">The march</a>'s goal was to draw attention to the federal government’s inaction in confronting the AIDS crisis and the Supreme Court’s 1986 ruling upholding Georgia’s anti-gay law. According to NCOD (National Coming Out Day), protesters flooded D.C. over a five day period and demanded legal recognition for gay and lesbian couples and the funding for AIDS research. The march also marked<a href="https://19thnews.org/2021/10/the-history-of-national-coming-out-day-contains-both-pride-and-pain/" rel="noopener noreferrer" target="_blank"> the unveiling</a> of the AIDS memorial quilt, a massive patchwork honoring those lost to the virus, and at the time an unprecedented show of support for gay rights.National Coming Out Day is meant to be a celebration of individuals within the LGBTQA+ community who openly identify as lesbian, gay, bisexual, transgender, non-binary, queer, asexual or intersex. It’s a day for honoring the act of coming out, and all the hopes, fears, dreams and expectations for the future.Whether it is at home, work, or in your community, coming out is the first step to living authentically. Scale is committed to creating a culture of inclusion that supports all Scaliens – no matter how they identify.We want to express gratitude to the Scaliens who have opened up to share their stories for this year’s National Coming Out Day:<h2 style="text-align: center;">Abe</h2>I grew up in a town of 1300 people in the middle of Southern Illinois. Where the pressures of never coming out and the expectations of settling down were standard.We moved from Missouri to Illinois around that time to help my grandma after my grandpa had passed suddenly of cancer.&nbsp;She lived on a farm in the middle of nowhere and that’s where me and my twin brother, along with an older brother and sister stayed for a few years until we moved to another house in the country when I was 9. During my tenure in grade school, there were a lot of slurs thrown around by kids who didn’t really understand what they meant.&nbsp;I never used them personally, because that's not who I’ve ever been. But I know if one of those slurs was directed at someone, it was a really bad thing and meant that kid was now viewed as less than a human.&nbsp;So I tried to fly under the radar.&nbsp;In middle school, I tried dating girls, because that was what you were supposed to do: date women, get married and have kids. I even pined after my best friend since the 4th grade until we were out of high school, but she never really returned the way I felt.&nbsp;When I got to high school, a kid in the grade above me was the first openly gay person I’d ever met.&nbsp;I admired the heck out of him for doing something I thought I could never do, and also didn’t understand why he’d want to subject himself to the small-minded mentality of the town we had grown up in.&nbsp;&nbsp;I finally graduated high school in 2000 and went to a small community college over by Centralia, IL.&nbsp;While going to college there, I was working at a gas station and an attractive male customer came in…that’s when I had my “holy CRAP, I’m gay” moment. I think many of you know what I’m talking about: the time you finally admit to yourself that you’re definitely not straight.&nbsp;I met the person who would become my future wife on the first day of freshman orientation.&nbsp;We were inseparable. We did everything together, enjoyed all the same activities like driving around, singing songs on the radio, and just enjoyed being around each other.&nbsp;After a few years, I thought to myself, “If there was anyone you could make a straight life with, it would be this woman.” So, I did what anybody who came from a town of 1,300 people would do: I asked her to marry me, driving down IL-4 one night.&nbsp;I was between Stelleville and Sparta when I popped the question…while driving…and didn’t have a ring. We were married on November 6th, 2004 at what was supposed to be a beautiful ceremony outside her parents house.Even before I had admitted to myself who I truly was, I had explored being with men.&nbsp;With that being said, a few months after I was married, I met another known gay person from my hometown (who had graduated high school right before I had entered) and struck up a friendship with him.&nbsp;Problem was, I ended up kissing him after the friendship started.&nbsp;Now this was the first time I had ever kissed another man, and my whole life changed.&nbsp;I couldn’t eat and seriously lost 40 pounds in 2 weeks.&nbsp;I couldn’t sleep, because I just thought I was always going to be married to my wife and now that I had kissed another man, I knew I couldn’t go on living as a straight man.So one night, I came home after a late shift, woke my wife up and told her the news, “Honey, I don’t know how to tell you this, but I’m gay.&nbsp;Here’s how it happened [see story above] and I’m truly sorry.&nbsp;I thought I could be straight and I just can’t be anymore.”&nbsp;There was a lot of crying that night between the 2 of us, a lot of discussion about the future and what it looked like for us, and how to start tomorrow off. I ended up coming out to everyone else that was important around me for the next few weeks and really received love and support from 99% of my family unit (I’ve always had the philosophy that family is the ones you choose as your family, so that doesn’t just mean people I’m related to).&nbsp;We ended up selling the house and 1 year and 1 month after we were married we officially signed the divorce papers.&nbsp;I still look back on that night with awe on how I ever had the courage to do it.&nbsp;I know I hurt some people, especially my ex-wife, but we still talk to this day and joke about that time in our life. She’s remarried with a beautiful daughter and I’ve even met her new husband.&nbsp;I’ve since moved to St. Louis where I try to be a beacon in the community, including heading volunteer efforts for an LGBTQ+ group I’m a member of, recently chairing a committee that helped raise $1500 for The Metro Trans Umbrella Group (MTUG) in one night.&nbsp;Yes, it was hard to come out, but I never liked closets anyway.&nbsp;The sunshine is so much better outside of them.<h2 style="text-align: center;">Willow</h2>I knew from an early age that I was gay and transgender, but it took me many years to come out. The social stigma felt very high. Gayness was used as an insult, and transgender people were a bad punchline at best.It took me 14 years of growing up to decide that I didn’t really care about the social stigma. During that time, I had focused on becoming an engineer and a soldier, stereotypically male professions. While deployed in Afghanistan, I was struggling to understand why I felt more depressed about the thought of returning home than staying in theater, and I realized that I was living my life inauthentically in a way that was leaving me devastatingly unhappy.When I got back home, I came out as queer and made plans to leave the military. In 2014, transgender people were still barred for military service, and while it seemed like that might change, I figured it would be a messy policy process with too much back and forth for me to want to live through (unfortunately this turned out to be correct).&nbsp;The first person I came out to as transgender was my long term girlfriend, who was incredibly supportive and caring. Next, I came out to my parents who said they loved and supported me and promised to assist in any way they could with transition. My immediate supervisor was also supportive, which made managing my exit from the military much easier and simpler. Even the most supportive coming out was very hard for me psychologically: it felt like a terrible admission of fragility and stigma. How much more difficult for those without support?I left the military, went back to school, and changed careers and genders. It was very challenging emotionally, but bluntly far easier for me than most other transgender people as a result of relative socioeconomic privilege and education, and due to the supportive friends and family around me. For this reason, it has always been important to me to be out as queer and transgender in my social life and business life. I do not treat these aspects of who I am as secrets, and I hope that being out makes coming out less of a burden for other people over time.<h2 style="text-align: center;">Richard&nbsp;</h2>I knew I was gay years before I met an openly gay person. I was 11 years old and watching TV when all of a sudden I found myself attracted to a male actor. For whatever reason, in that moment, I had no doubt that I was gay. I didn’t want to be - “gay” was a prejorative even among 5th graders in midwest suburbia - but it was a fundamental truth, as undeniable as 2+2=4.It never occurred to me that I could hide this forever. Living in the closet requires telling dozens of lies daily. Some are overt. Many are subtle, a slight blur of the truth. It’s an exhausting existence, one that I personally was not capable of. Rationally, I knew that I had nothing to be ashamed of. But as much as that made logical sense, it was still emotionally terrifying - would I lose my friends? Could I have kids and build a family? Could I have a “respectable” career and be a “respectable” member of society?&nbsp;I’m a big believer in “getting your reps in” - you need to practice in order to build muscles, whether they’re physical or emotional. Coming out isn’t a one-time thing. You come out over and over again. To your friends’ parents. To new colleagues. To old friends you didn’t really keep in touch with, and you aren’t quite sure if the gossip reached them. It took me a few years to learn how to come out without showing my fear externally, and a few more to not feel it internally either. It took dozens of reps to rewire the part of my brain that would go into overdrive - that flight or fight response that kicks in when you subconsciously perceive that you’re in danger.As a teenager, coming out was the hardest thing I had ever done in my life. Arguably, it still is. There were no openly gay students at my high school. There were no openly gay men in my college fraternity. It was terrifying, but what was once a source of terror is now a source of pride.One of my fondest memories was visiting my brother in 2012. I was 20 years old, and he had joined some small startup called Dropbox in some far away place called San Francisco. I was eating lunch served by company chefs in the office cafeteria - that in itself was a novel concept - and overheard a same-sex couple, surrounded by friends and colleagues, talking about their baby. My mind was blown by how at ease they were, and how “normal” it all felt. At that moment, I really felt that I could both be gay and have the foundational things I wanted - friends, family, and a career. My dream was a possibility, and it was up to me to make it a reality. For this, I will always be grateful.During my lifetime, the world has overall become a more accepting place. I’m optimistic that, even if there are some bumps in the road, we’ll continue to make progress, and more and more people will have the freedom to live authentically.&nbsp;

The celebration started in 1988 on the first anniversary of the second National March on Washington for Lesbian and Gay Rights in October 1987. 

National Coming Out Day is a celebration of individuals within the LGBTQA+ community 

Scale Out: National Coming Out Day 

Model evaluation is one of the most important prerequisites prior to shipping an ML model. If you don’t properly evaluate your model, performance regressions might appear in production and have a direct and negative impact on your business. For example, if a faulty pedestrian detection model is deployed onto an autonomous vehicle, it can potentially cause an accident. In its most basic form, model evaluation is very straightforward, but as we discussed in our <a href="https://scale.com/blog/find-best-model" rel="noopener noreferrer" target="_blank">previous blog</a>, approaches that are too simple aren’t really reliable. Therefore, most ML teams aspire to maintain a mature model evaluation pipeline in order to systematically understand model performance. A mature evaluation pipeline covers the entire lifecycle of ML development:<ol><li>Model training and re-training</li><li>Model testing and validation</li><li>Model performance monitoring in production&nbsp;&nbsp;</li></ol>To cover all these aspects, ML teams need sophisticated tooling, which is why we assembled a list of 4 MLOps tools that we believe will help you build a great model evaluation pipeline:Weights &amp; Biases<img src="https://lh5.googleusercontent.com/FT9Rlbx977RDuvofnHrtTOxQPjRAjHtncmZQJZel9sDeCRiRYgdle8ubOonzVkVGgkxhxuv_7J5oSyXSSMpU2yZa-Vkv4zYxXibg2Z-nB42bjTd3hl4O8PaZc_Vpluk7S8Wsw5M8GpxNOw4mep2TOiqYX08B0haUhu-_YqmI9YXes0WfSIZAJV1KGg"><a href="https://wandb.ai/site/experiment-tracking" rel="noopener noreferrer" target="_blank">Source</a>Weights &amp; Biases (W&amp;B, for short) is a great tool for logging information and analyzing performance during model training. It is designed to track standard training metrics like loss, training dataset (“train”) accuracy, validation (“val”) dataset accuracy, GPU utilization, and more.&nbsp; W&amp;B will also log your raw data like images or point clouds along with their matching predictions like bounding boxes, cuboids, and segmentation masks. Moreover, W&amp;B supports experiment tracking. Users can log training run configurations, including sets of hyperparameters, and use these to reproduce the experiment later. Already widely adopted in the ML community, W&amp;B is a more powerful cloud-hosted alternative to TensorBoard, especially if teams want to converge on a collaborative tool.COCO Eval API<img src="https://lh5.googleusercontent.com/SXczlMWaBJoU7IofJKyyrJoP1yYnfo_m7VvAY55w_eZMU-tvP9Bz5Q69U87f18Qv_6gvZ7FGWLjdnH2PZqfJm1bp9BxwyqQvuas3FT-hxgZkBwExjg_cifCN8noswnenhVnDsPjo1P8inay5wUrFAMHhFhvTNiCaITChXgcP8-5DCF-a-g036X6mjA">While there are a lot of tools available to log training and evaluation metrics, COCO’s eval API is among the handful that provide these metric definitions and implementation out of the box. Released as a toolkit for interacting with the MSCOCO dataset, the COCO eval API has now become the standard tool for many ML developers working on computer vision use cases like object detection. The API supports the calculation of IoU, precision, and recall on bounding boxes, segmentation masks, and keypoints. It also allows users to aggregate these scores—both per item and per class. Although very bare bones, the COCO eval API is a handy tool for obtaining standard metrics.Scale Validate<img src="https://lh6.googleusercontent.com/H5OAAYhwy544Pj2iFx8yuDCeKxPJ7phxvazeRPgDgYI5s8tUTHra66cuqLzgxHTcCT6qOJuKG8Wd3DQTnkP754VhCekZMV2wqQXZxgUfp2oTWNIhwnGtPIp4a4g5gUphE41hhAJ_X3DaJU-hx7bNbJ9I3LfALBWJg7nrFGslKQkcN0-i5G11EjvPsA">While metric computation and logging are essential, they are not sufficient to uncover the true performance of the model prior to deployment for reasons such as data inequality and comparison complexities (<a href="https://scale.com/blog/find-best-model" rel="noopener noreferrer" target="_blank">you can learn more here</a>). ML teams frequently tackle these challenges using Scale Validate, which provides a structured framework for testing and comparing ML models. Validate allows you to:<ol><li>Compare models on a rich set of pre-computed metrics e.g. precision, recall, confusion matrix, IoU or mAP.</li><li>Find underperforming slices through automatic clustering based on image embeddings and metric scores&nbsp;&nbsp;</li><li>Set up tests on mission-critical scenarios to track progress on metrics of choice over time and catch regressions.</li></ol>Through these strategies, Validate helps dissect obscure aggregate metrics into more granular insights and ensures that the model being shipped performs better not just on the aggregate but also on the more critical and rare long-tail situations.&nbsp;Why Labs<img src="https://lh5.googleusercontent.com/AO7znH2bIj4K__AVI3X-NGBdBqbpMhWXE_tUQ3Q-3XM_n4Xybl-1DG7vsn2BTWbsPJpn3LqQgMif9Smq-GOsl3tD5HyZjRmurRj8JgErcIVp2W1IC_ZBO54wyiJfPhg8rJfzsAaZG18PUXy7XEmDOtn-7W7lWp-XXQSm7nFzYzeDjlV7zOtwYva1PA">After a model has been deployed to production, it is crucial to keep monitoring its performance to ensure that the model performs as expected. WhyLabs is an observability platform designed to monitor ML model performance degradation. Users can create monitors to target a particular performance metric—typically, precision, recall or F1 score. Once the monitors are set, WhyLabs tracks the metrics relative to baselines set by the user. These baselines can be based on a dynamic range, like an average metric value over a historic time interval or a static fixed value. Using these monitors, users can also set corresponding actions if the value crosses certain thresholds, like alert messages. WhyLabs can be deployed in production cloud environments including AWS and GCP, or on-premises.&nbsp;Great AI applications require dependable ML models and an ML model’s dependability can’t be ascertained without foolproof evaluation. Using these tools, ML teams can build a reliable evaluation pipeline to uncover true model performance and avoid any regressions in production. <a href="https://dashboard.scale.com/nucleus/scenario-tests?utm_source=blogpost&amp;utm_medium=website&amp;utm_campaign=top_eval_tools" rel="noopener noreferrer" target="_blank">Get started with Scale Validate</a> today to uncover and fix regressions over critical scenarios.

Model evaluation is one of the most important prerequisites prior to shipping an ML model.

Top 4 Tools to Build Your ML Model Evaluation Pipeline

Scale is the AI Infrastructure provider of record for the commercial autonomous vehicle (AV) industry, powering nearly every American commercial AV company’s perception and AI efforts. This experience informs our approach to defense autonomy applications.&nbsp;The development of successful Army Autonomy programs largely hinge on two factors:&nbsp;<ol><li>AI Data Management. Deliberate collection, curation, processing and enterprise management of vast amounts of real-world, AI-ready training data.</li><li>Enterprise ML Ops Infrastructure.&nbsp;Comprehensive, end-to-end infrastructure to develop, train, test, validate, deploy, and continuously iterate on artificial intelligence and machine learning (AI/ML) models.</li></ol>We currently execute billions of <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a> per month, HD map 3,800 kilometers of roads per week, and are the industry leading AI perception data solution across the globe.&nbsp;Scale partners with self-driving vehicle companies training systems from small delivery cars to long-haul trucking and up-armored military vehicles. Our partners include Kodiak Robotics, Voyage, Optimus Ride, Toyota Research Institute, Embark, General Motors, Honda innovations, Oshkosh, and Nuro - the first autonomous ground delivery vehicle to receive <a href="https://scale.com/customers/nuro" rel="noopener noreferrer" target="_blank">DoT approval</a> for public roads.&nbsp;Additionally, Scale is helping to develop autonomy in other unique domains, including (but not limited to) maritime vessels (<a href="https://scale.com/customers/sea-machines-robotics" rel="noopener noreferrer" target="_blank">Sea Machines</a>), autonomous drones (<a href="https://scale.com/customers/skydio" rel="noopener noreferrer" target="_blank">Skydio</a>), and autonomous warehouse systems (<a href="https://scale.com/customers/otto-motors" rel="noopener noreferrer" target="_blank">OTTO Motors</a>).&nbsp;Through our in-house capabilities, direct partnerships and an extensive subcontractor network, Scale collects, curates and <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotates</a> more ground-based perception data than any other company in the world.&nbsp;Synthetic data can be created and used to train against unique edge-cases that Army autonomous vehicles would face in the field, and data from various sources can be fused and mapped for unparalleled autonomous perception.&nbsp;&nbsp;&nbsp;We are a $7B company with real-world applications at level 9 technology readiness. Scale has over 700 employees focused on cutting edge ML development, human-machine teaming, content and understanding, and development of novel applications for machine learning in the commercial and federal sectors. Our Modular Open System Architecture reduces cost and time to deployment, increases opportunities for iteration and upgrade, allows the government to pick and choose the tools and data it needs, and the data rights remain with the government.&nbsp;Scale supports the US Government in upholding democratic ideals. We’ve already done it with many other DoD organizations.Come talk to us at the Association of the United States Army (AUSA) conference in Washington, DC from 10-12 October, 2022 (booth 4032), check us out on <a href="https://www.linkedin.com/company/scaleai/" rel="noopener noreferrer" target="_blank">LinkedIn</a>, or <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">contact us directly</a>!Jonathan Hudgins is a member of the Federal Team at Scale where he focuses on bringing Scale’s proven commercial solutions to solve Federal Government problems. Jonathan is a former active duty Air Force officer and F-15E Fighter Pilot.

Through our in-house capabilities, direct partnerships and an extensive sub-contractor network, Scale collects, curates and annotates more ground-based perception data than any other company in the world.

Our approach to defense autonomy applications

Scale AI: The Data Platform for Autonomy

Any ML developer who has trained an ML model multiple times e.g. with different hyperparameters, and tried to figure out which iteration is better, is already familiar with the basics of model comparison. Before her model can deploy to production, an ML developer needs to be confident that it would perform better than the previous model. However, this is easier said than done. Comparing models with the intent of figuring out which one is better is a complex task. Most ML developers take a rather rudimentary approach, which creates a significant risk of shipping the wrong model.In this blog, we will go through the core challenges associated with model comparison and discuss strategies to mitigate them. Before we do, though let’s take a quick look at the basic approach adopted my most ML developers:<h3 style="text-align: justify;">The basic approach</h3>Let’s say we have recently trained two versions of an object detection model with a different learning rate and now want to find out which one is better. Here’s what that process looks like for most ML engineers:<ol><li style="text-align: justify;">Choose a standard aggregate metric e.g. mAP (mean average precision), <a href="https://en.wikipedia.org/wiki/F-score" rel="noopener noreferrer" target="_blank">F-1</a> score, precision or recall&nbsp;</li><li style="text-align: justify;">Apply <a href="https://towardsdatascience.com/non-maximum-suppression-nms-93ce178e177c" rel="noopener noreferrer" target="_blank">non-max suppression</a> to reduce the number of candidate predictions, ideally leaving the best one accessible and viewable to the user</li><li style="text-align: justify;">Match each of the resulting predictions with ground truth using intersection over union</li><li style="text-align: justify;">Set minimum IoU and confidence scores thresholds to obtain class confusions</li><li style="text-align: justify;">Using these confusions—True Positives (TPs) &amp; False Positives (FPs)—calculate each model’s aggregate metric&nbsp;&nbsp;</li><li style="text-align: justify;">The candidate model with the highest aggregate metric score is best and the one you’ll want to deploy&nbsp;&nbsp;</li></ol> <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F17406fbd379046e08c2882ab8590acb4"> <img src="https://lh5.googleusercontent.com/AYGlxcGUVLCh48j-UXIyixhm5qtp4hZTrGtycJQm6ejAuV-Qx3z_ibyj_nWS-ACIEsTmLJ4hS2xBAoQy1Xgjrndgzoxud3o2wVT3gO7c4NYBrAgpq1tYASskbRH_0oZtzTTr-NHC1Bf_lS62mulaY_u3Snf2KrxtQ03wIFubYFLhbVku1hPB9sqOZA">While such a one-dimensional and aggregate comparison is helpful thanks to its simplicity, on its own it’s definitely not sufficient to identify the best model. An aggregate approach obfuscates performance over critical rare cases by assuming all data is equally important. Finding the best model is a complex multidimensional process and more often than not relies on the developer’s subjective opinion. Over the past years, we have worked closely with many ML teams on this topic and have identified strategies to systematically find the best model. It all starts with finding the right metric: <h3 style="text-align: justify;">💡Deciding on the right metric&nbsp;</h3>More often than not, ML engineers choose the standard metrics—for example mAP (referenced above) for object detection. However, metric selection is one of the most important decisions when evaluating and comparing models. The metric must capture the nature of the application and the business expectations from the model. For example, if you are working on a cancer detection problem, you would want to detect as many cancer cases as possible even if it means in a few scenarios, you classify a healthy person as having cancer. So “Recall” would be a great metric. However, if you are performing email spam detection then it’s acceptable to miss a few spam emails but not okay to classify a legit email as spam. So “Precision” would be the metric of choice. Typically, you’ll need to make a trade-off between precision and recall. In these situations, tools like the PR (precision-recall) curve or <a href="https://en.wikipedia.org/wiki/F-score" rel="noopener noreferrer" target="_blank">F-scores</a> can help better capture model performance.&nbsp; To get a feel for these metrics, try computing a sample metric on a slice of data in <a href="https://dashboard.scale.com/nucleus/scenario-tests?utm_source=blogpost&amp;utm_medium=website&amp;utm_campaign=best_model" rel="noopener noreferrer" target="_blank">Scale Validate</a>.<a href="https://scaleai.wistia.com/medias/9sp4zf58r3" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/9sp4zf58r3</a> <h3 style="text-align: justify;">🔎 Finding the optimal operating point</h3>Operating point is just another name for input controls which determine confusion calculations e.g. for object detection the operating point is a combination of minimum IoU and Confidence thresholds, if the thresholds are 0.5 then the evaluation logic will only consider objects with &gt;0.5 confidence and &gt;0.5 IoU. So you would want to ensure that these thresholds are set such that your model is giving the lowest number of confusions and the highest performance e.g. the highest F1 score. However, the extent to which you can reduce or increase these thresholds depends on your application. For example, you don’t want a low IoU threshold for object detection because that would mean large errors in location &amp; size of the detected objects. Therefore, you need to find the application constraints first and then find the optimal point for each candidate model within those constraints. To find the optimal point you can traverse over each sensible range of these input controls and look at the resulting performance metric. Once you have the optimal operating points you can do an apples to apples comparison of the aggregate scores. Try varying the IoU &amp; Confidence thresholds of this <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=insights&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg" rel="noopener noreferrer" target="_blank">sample model on Scale Nucleus</a> and see the effects on the confusion matrix and PR curve.&nbsp;<a href="https://scaleai.wistia.com/medias/x2yaorc7f0" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/x2yaorc7f0</a> <h3 style="text-align: justify;">🧮 Complexity of multiple metrics</h3>In most instances it is hard to capture model performance with a single metric. For example, for obstacle detection you don’t just want to detect an object but also be very accurate with the location and size. Therefore, on top of an accuracy metric such as recall, you would also want to track average IoU to check for the tightness of the predictions being produced. The challenge of doing model comparison with multiple metrics is that no model might perform the best on all metrics. To tackle this problem, the simplest method is to do a weighted aggregation of these metrics and obtain a single score. This is exactly what the F-scores do for the metrics of precision &amp; recall. If you want to assign equal importance to both metrics you can compute F1 score, otherwise you can use F0.5 or F2 scores to weigh one metric more than the other. You can read more about F-beta scores <a href="https://machinelearningmastery.com/fbeta-measure-for-machine-learning/" rel="noopener noreferrer" target="_blank">here</a>. Try <a href="https://dashboard.scale.com/nucleus/model-comparison?analyze_dataset_id=ds_bwkezj6g5c4g05gqp1eg&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;analyze_compare_model_id=run_cabt1b5e3xf005r21sxg" rel="noopener noreferrer" target="_blank">model comparison over multiple metrics</a> on Scale Validate.<a href="https://scaleai.wistia.com/medias/g5ie6sy2bl" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/g5ie6sy2bl</a> <h3 style="text-align: justify;">📊 Dealing with data inequality&nbsp;</h3>Real world data is never equally distributed or equally relevant. Some subsets or slices are just more important than others for the model in question. And more often than not these high importance subsets are underrepresented and hence small in size. The way aggregate metrics are calculated, they assume that all data is equally important and obscure performance on rare important samples. For example, a pedestrian detection model can have an mAP of 0.9. But that 0.9 can be very misleading if the examples it fails on are the ones where pedestrians are on a crosswalk! Similarly when comparing two models, the aggregate performance might go up but that doesn’t mean that the new model is better. To address this it is important to first identify these longtail situations and curate small subsets which act as data unit tests similar to unit tests for software. Then models can be compared with regards to how well they perform over these unit tests in addition to the aggregate metric. For example, model B is only better than model A if it improves on the aggregate and doesn’t regress on any of the data unit tests. Read more about data unit tests <a href="https://scale.com/blog/testing-models-validate" rel="noopener noreferrer" target="_blank">here</a> and try to compare models on specific unit tests in Scale Validate.<a href="https://scaleai.wistia.com/medias/0kekfm8stb" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/0kekfm8stb</a> <h3 style="text-align: justify;">To summarize:&nbsp;&nbsp;</h3><ol><li style="text-align: justify;">Choose metrics which are representative for your application</li><li style="text-align: justify;">Carefully analyze and define thresholds in your models</li><li style="text-align: justify;">Ensure that critical subsets of data have an impact on the evaluation score</li></ol>We have built Scale Validate with these strategies in mind. You can try it for free on your data or an open source dataset <a href="http://scale.com/validate?utm=n" rel="noopener noreferrer" target="_blank">here</a>.

Scale Nucleus

 Model comparison can be hard—here is how you can find your best model!

Finding Your Best Model Is Tricky—Here's How to Do It!

<h3>A little about myself</h3>I’m Nikash, I’m a Software Engineer at Scale. I joined in August 2021 on the mapping team and am now working on content understanding.&nbsp;I graduated from the University of Virginia in May 2021, where I studied computer science and cognitive science. Before graduating, I had previously interned in biotechnology at Benchling, autonomous vehicles at Uber ATG, and fintech at Capital One.&nbsp;A fun fact about me is that I love Cuban food! Guava cheese pastries, cafe con leche, Cuban sandwiches, fried plantains…you name it. Check out Portos in Los Angeles or Colada Shop in DC if you ever get the chance.<h3>Why Scale</h3>Scale has been on my radar since 2019 when I was interning in the self-driving space. After finding Scale on <a href="https://www.ycombinator.com/topcompanies/" rel="noopener noreferrer" target="_blank">a list of top YC alumni</a>, I started reading more about the work they do with autonomous vehicle companies and other major tech players. I was especially fascinated by Scale’s light-speed growth and impressive customer list.Out of college, I knew I wanted to work at a relatively smaller company where my work would have incredible impact. My good friend <a href="https://www.linkedin.com/in/rashidlasker/" rel="noopener noreferrer" target="_blank">Rashid</a> was already working at Scale and told me about the super cool projects he was working on. I was deciding between a few similarly-sized startups and ultimately chose Scale because of the breadth of industries they work in; from autonomous vehicles to social media leaders and much more.&nbsp;<h3>My experience on the Mapping team</h3>I started Scale on the sensor fusion team where I worked on mapping for self driving cars. Autonomous vehicles rely on high-definition maps for perception, localization, and motion planning. Think of mapping as 2.5-dimensional labeling. AV customers send us LiDAR point clouds, which are collections of points representing scenes in the 3D space. We pre-process these point clouds and flatten them into a 2D or “top-down” view while preserving information like height. Then, we can draw road features like lane lines and traffic lights on this “top-down” view while being able to project them back into the 3D space.&nbsp;While on the mapping team, my first project was improving the quality of the maps we built, using computational geometry linters to validate map features. With these linters, I built a series of autofixes or autochecks to serve as a human-in-the-loop automated solution to programmatically fix quality issues of maps. This was super engaging and technically challenging work; I found myself using trigonometry and graph theory concepts I learned at school directly in the code I was writing! My manager even once recommended I read a computational geometry textbook he studied in order to solve a particularly difficult problem I was facing. It’s definitely quite rare to apply advanced math concepts from academia at work, and so I loved that this was my first main workstream at Scale.I also primarily worked on algorithmically annotating semantic or “derived” zones of roads. Historically on our team, we focused on labeling observable features or pseudo-observable features on the map. That said, we knew our customers were having lots of trouble using these observable features to derive semantic regions and so we took a stab at solving this internally. These “derived” regions could include lanes, intersection areas, and other regions that are algorithmically developed but are based on visible road features. Ultimately, our proposal anticipated a massive pain-point that our customer had which led them to require these derived annotations in the maps we built. At Scale, we serve as “thought leaders” and technical partners for our customers, anticipating and solving their hardest open-ended challenges.<h3>My experience on the Content Understanding team</h3>In April, our leadership reached out to me asking if I’d like to participate in a new 6-month rotational engineering program called “Nighthawks”. As a part of the group, I’d build out new products from 0 to 1 and tackle the most critical and high-impact customer projects. I joined the product incubation and content understanding team, and since joining I’ve owned critical pilots with demanding timelines for a range of fascinating clients, including social media platforms and an NFT marketplace. On these projects, I’ve analyzed trends in social media and worked in detecting fraud, trust and safety issues.Easily the most interesting aspect of the Nighthawks program has been participating in Go-to-Market, and learning how to interact with prospective customers. Participating in conversations with our sales teams, I’ve learned just how tactical and strategic Scale is when landing major deals, and I’ve observed a side of B2B that I hadn’t been exposed to before. Through Nighthawks, I’ve had the chance to demonstrate Scale’s offerings directly to many of these clients, which has taught me a ton about customer-oriented engineering.&nbsp;<h3>Life at Scale</h3>Going into Scale, I had participated in a few internships where I built relatively siloed products, but I had never experienced true customer interactions and owning outcomes. At Scale, one’s responsibilities extend far beyond those of a vanilla engineer. One of our credos is “ownership is the job”; to succeed in delivering an outcome you’re required to participate in product-oriented discussions, be embedded in operations, and communicate directly with customers. The life and work of a Scale engineer is far more diverse than any other role I’ve been in prior to joining, which contributes to a vibrant and exciting work experience.

Hear from a Software Engineer at Scale

For anyone following AI or machine learning trends in recent weeks, text-to-image generation has taken the spotlight by force. A never-ending stream of synthetically-generated (yes, algorithm-generated!) images is populating social media feeds. This was very different just a couple of months ago, before an invite only beta version of <a href="https://openai.com/dall-e-2/" rel="noopener noreferrer" target="_blank">Open AI’s DALL-E 2</a> was opened to the public in July. With DALL-E available via signup and waitlist, and now with <a href="https://github.com/CompVis/stable-diffusion" rel="noopener noreferrer" target="_blank">Stable Diffusion</a> released to all as public open-source software, developers have been prolific and building and hosting an array of online image generators. Perhaps the users of these image generators are even more prolific than their authors—libraries of millions of generated images are swiftly gaining popularity, making it easy to browse other people’s creative ideas and find inspiration. We’ve put together a list of the 5 best, openly available online AI image generators using <a href="https://github.com/CompVis/stable-diffusion" rel="noopener noreferrer" target="_blank">stable diffusion</a> to help you jump right into the mix.<h3><a href="https://beta.dreamstudio.ai/dream" rel="noopener noreferrer" target="_blank">Dream Studio</a></h3>This is the foundational tool released by the company that created Stable Diffusion, Stability AI. The basic version requires a login, but is free to use. The generator offers a ton of settings, including the model version, Cfg scale (how close the image will be to your prompt), steps, number of outputs, image dimensions and more. You can start both from scratch or with a seed image of your choice. Conveniently, Dream Studio keeps a history of all your creations and lets you download the full resolution image. There is also a guide on how to write the best prompts so that the generator achieves the best results.<h3><a href="https://replicate.com/stability-ai/stable-diffusion" rel="noopener noreferrer" target="_blank">Replicate</a></h3>Replicate is a small startup on a mission to make machine learning models easier to use for everyone. They provide a service that helps you use previously uploaded models or upload your own model to run inference on demand, instead of spinning up your own server. With the wave of interest in stable diffusion, they have created a useful web app to use the model in order to generate images as well. Signup is required, but otherwise it is free to use. To date, the model has generated more than 5 million images!<h3><a href="https://creator.nightcafe.studio/stable-diffusion-image-generator" rel="noopener noreferrer" target="_blank">NightCafe Creator</a></h3>NightCafe is a company whose mission is to democratize the creation of art using technology. They have been going strong since 2019, but have recently gained huge momentum with the advancements of text-to-image models. A simple text field invites you to enter your prompt, but advanced customizations are available. You can generate up to 5 images for free, but have to buy credits afterwards. NightCafe also offers other creation models besides stable diffusion, such as VQGAN+CLIP, which was their most popular before 2022. They offer a huge collection of artworks created by other users, a Discord server and a Subreddit for discussions.<h3><a href="https://www.phase.art/" rel="noopener noreferrer" target="_blank">Phase.art</a></h3>This project is a simple text-to-image generator using stable diffusion, with only a simple input for a prompt. There are suggested categories and tokens that can be used to compose the prompt and get inspiration. Creation requires login, which gives you 10 credits (equivalent to 10 images generated) and also enables you to revisit your previously generated images. After the first 10, you have to buy additional credits to keep creating. Phase.art offers a Discord server to connect with fellow text-to-image artists.<h3><a href="https://www.craiyon.com/" rel="noopener noreferrer" target="_blank">Craiyon</a></h3>This AI image generator was previously known as DALL-E mini, and became wildly popular as the original DALL-E by OpenAI is not yet openly available to everyone. The authors, Boris Dayma and Pedro Cuenca, attempted to implement a similar image generator, but using an open source model. This is actually not a stable diffusion model and the results are sometimes a bit less polished (especially for faces) when compared to other image generators. However, the service is entirely free and enables you to download your creations, even if only as a screenshot and not in full resolution.<h4>Not sure how to get started or what prompts to use?</h4>Fortunately, with web apps like <a href="https://lexica.art/" rel="noopener noreferrer" target="_blank">Lexica.art</a> and <a href="https://arthub.ai/" rel="noopener noreferrer" target="_blank">ArtHub.ai</a> there are several fast growing libraries of images that other people have created using stable diffusion including the relevant prompt they used in the AI image generator. If you’re in lack of ideas, go ahead and browse the prompts that other users submitted to get started!If you'd like to learn more about Stable Diffusion and other generative image models, check out our <a href="https://scale.com/guides/diffusion-models-guide" rel="noopener noreferrer" target="_blank">Guide on Diffusion Models here</a>.

5 tools to assist you in generating beautiful art from text prompts with ML models.

5 open, freely available AI image generators to get you off the ground.

How to Make Your Own AI-Generated Images

<a href="http://wwww.scale.com/" target="_blank" rel="noopener noreferrer">Scale AI</a>, the data infrastructure for AI, today announced the achievement of the &ldquo;In Process&rdquo; designation for its AI Data platform by the Federal Risk and Authorization Management Program (FedRAMP), a crucial milestone in the FedRAMP certification process. This designation indicates that Scale is on track to achieving a FedRAMP authorized product within the next 12 months.&nbsp;The AI Data platform is seeking authorization at the FedRAMP High Impact Level, enabling Scale AI to support the U.S. government&rsquo;s most sensitive data and missions.&nbsp;
Scale serves customers of all sizes across a wide range of industries. Reaching this milestone on the path to authorization is invaluable to Scale&rsquo;s ability to enable federal, state and local governments in the adaptation of AI applications.&nbsp;
FedRAMP is the U.S. Government&rsquo;s standardized approach to providing security authorizations for the adoption and use of cloud services by the federal government. The government-wide program&rsquo;s goal is to accelerate the use of secure cloud technologies by government agencies. The &ldquo;In Process&rdquo; designation is an important step towards Scale AI&rsquo;s full FedRAMP authorization at the FedRAMP High Impact Level.
Scale&rsquo;s commitment to security is one reason why leading ML teams from private sector organizations and the U.S. Government rely on Scale AI to accelerate the development of their AI applications. If you&rsquo;d like further details, please contact us at fedramp@scale.com.
&nbsp;

<a href="http://wwww.scale.com" rel="noopener noreferrer" target="_blank">Scale AI</a>, the data infrastructure for AI, today announced the achievement of the “In Process” designation for its AI Data platform by the Federal Risk and Authorization Management Program (FedRAMP), a crucial milestone in the FedRAMP certification process. This designation indicates that Scale is on track to achieving a FedRAMP authorized product within the next 12 months.&nbsp;The AI Data platform is seeking authorization at the FedRAMP High Impact Level, enabling Scale AI to support the U.S. government’s most sensitive data and missions.&nbsp;Scale serves customers of all sizes across a wide range of industries. Reaching this milestone on the path to authorization is invaluable to Scale’s ability to enable federal, state and local governments in the adaptation of AI applications.&nbsp;FedRAMP is the U.S. Government’s standardized approach to providing security authorizations for the adoption and use of cloud services by the federal government. The government-wide program’s goal is to accelerate the use of secure cloud technologies by government agencies. The “In Process” designation is an important step towards Scale AI’s full FedRAMP authorization at the FedRAMP High Impact Level.Scale’s commitment to security is one reason why leading ML teams from private sector organizations and the U.S. Government rely on Scale AI to accelerate the development of their AI applications. If you’d like further details, please contact us at fedramp@scale.com.<h3>About Scale AI</h3><a href="https://scale.com/" rel="noopener noreferrer" target="_blank">Scale </a>accelerates AI development by providing the data infrastructure for AI. Its data-centric platform manages the entire ML lifecycle, from <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> and curation to model testing and evaluation, enabling any organization – from the world’s most advanced AI teams to legacy organizations – to develop and deploy impactful AI solutions. Scale combines ML technology with skilled human insight to ensure every AI application is built on a foundation of high-quality ground truth data. The market leader in the autonomous vehicle industry, Scale is trusted by the largest technology companies in the world, including Microsoft, Toyota, GM, Nuro, Samsung, Square, PayPal, SAP, Instacart, Pinterest, Brex, Flexport and U.S. government agencies like the U.S. Army and Air Force. Scale was founded in 2016 and is valued at $7.3 billion, backed by Founders Fund, Accel, Index, and Tiger Global.

Scale is on track to achieving a FedRAMP authorized product within the next 12 months

Scale achieved a crucial milestone in the FedRAMP certification process

Scale Achieves FedRAMP ‘In Process’ Designation for its AI Data Platform

Delivering a great buying experience makes all the difference in whether or not a user will convert to a customer. A core part of the eCommerce buying experience is creating an exceptional user journey from initial discovery to purchase. To learn more about accelerating the customer experience, we spoke to Nick Beighton, former CEO of ASOS, about his experience bringing a data-driven approach to online retail.&nbsp;<h2>Data Attribution as the Foundation for the Customer Experience&nbsp;</h2>For a brand to be successful, it needs to have agility in its business and technology to scale quickly to improve shopping experiences based on customer needs. At the core of agility for eCommerce companies, digitization is vital on both the front and back end. Nick explains,“Digitization is really important on the front end, but it’s also important on the back end in terms of how you digitize your processes, how you use data to drive greater personalization, and how you make experiences more efficient, faster, and intuitive.”&nbsp;Nick explains that digitizing the front end encompasses creating intuitive and personalized experiences. As customers engage with the digital offer, the experience becomes more customized, from ads to product recommendations. Scaling operational processes on the back end centers on strong data discipline. A key part of this is maintaining a large catalog of product attributes, which encompasses everything from physical characteristics such as size, color, and material to brand and product descriptions.“Digitizing the back end can only be done at scale through strong data attribution, strong control of your product master files, and the attributes within it. To attribute that data correctly, you can then turn your customer care from a cost center, to a profit center, by using the data to add on more sales and think differently about that interaction with the consumer.”By starting with strong data attribution, eCommerce companies can better connect marketing and SEO investments to the entire customer experience.<h2>Investing in Product Catalog Data at ASOS</h2>Nick applied a data-focused approach to scaling ASOS by first focusing on the search journey on the ASOS website. A core area that the ASOS team investigated was the differing conversion rate between customers who used the search bar to find a product and customers who used the website navigation: customers who used the search bar had double the conversion rate. As a result, the team explored how to maximize the search journey. One discovery was how product attribute data impacts the customer search journey.“We realized that the search journey didn’t have all the right data attributes to serve the most relevant search. So if you’re searching for men’s shoes, you might end up with women’s shoes too. We had to clean up the data on our main website to improve the search journey, and the recommendations served to the customers. That was one of the early ways we started investing in data attribution and product catalogs through the journey.”Nick spearheaded an ML team that maximized product attribution data, and the investment proved successful. By improving product catalog data, conversion moved from 2% to over 7% on average and quadrupled the number of major SKUs added to the site. Nick explains, “Data attributes and attribution of product information was critical to maintaining and improving that conversion while simultaneously adding more and more products.”&nbsp;Investing in product attributes proved instrumental at ASOS. The team connected robust product catalog data to maximize ROI and focus on conversion and customer experience. By building strong AI operational processes, the ASOS team optimized how they served customers.&nbsp;<h2>How eCommerce Companies Can Get Started with a Data-Driven Approach</h2>How do eCommerce companies get started with having strong data discipline? What are some of the best ways to implement these practices across an organization? Nick explains the critical areas for teams to focus on are investing in data attribution early, experimentation, and establishing product goals:<ol><li>Invest in data attribution: Strong product data attribution is at the core of improving the customer journey. ML teams can address long-tail product and content issues by investing in data attribution.&nbsp;&nbsp;</li><li>Experiment to get started: Don’t limit experimentation with tech solutions that only have strong business cases and ROI. You may not know what experiment will give you an exponential return, so it’s essential to have multiple experiments and tests in parallel and review the data to understand the impact.</li><li>Align on product goals: Investing in product catalog data and tying the data attribution goals to product performance metrics is crucial to transforming the business. By working closely with product teams, teams working in eCommerce can provide a direct correlation with internal metrics.&nbsp;</li></ol>Nick emphasizes the importance of data discipline is all about creating the best end experience for the customer,“A customer will rarely say, ‘I love your data attribution,’ right? She will say, ‘I feel the site gets me and gives me all the things I want when I’m searching for it.’ That’s the consumer mindset goal, and how to achieve it is through digitalization and data discipline.”&nbsp;At Scale, we believe investing in data is the key to unlocking success for<a href="https://scale.com/retail" rel="noopener noreferrer" target="_blank"> eCommerce companies</a>. If you’re interested in learning more about boosting your business’s conversion, speak to our team today.

Nick Beighton, former CEO of ASOS, discusses how he transformed the company from 200 million to 4 billion in revenue by investing in product catalog data. 

How Nick Beighton Transformed ASOS by Investing in Product Catalog Data

How to warm-start your computer vision project with these 10 robust public datasets prior to fine tuning on data for your use case.

How to warm-start your computer vision project with robust datasets prior to fine tuning on data for your use case.

The 10 Best Public Datasets for Object Detection in 2022

Roon guest-authors a post on large language models and the future of computing.

Text became the most common lever by which mankind would later induce computers to do their bidding.

Text Is the Universal Interface

Aerin is an engineering manager at Scale AI, leading a Catalog ML team. Below are Aerin's stories about what she does and why she loves working at Scale.<h2>What brought you to Scale?</h2>Before I joined Scale, I worked on a project called NL2SQL which stands for "Natural Language to Structured Query Language" at Microsoft. NL2SQL is a translation problem that automatically converts a user's spoken-language query into SQL.Just a few months after my team had started working on it, we were beating the SOTA (state-of-the-art) performance on research benchmarks. We achieved this by evaluating different ML model architectures and improving them. In other words, we somewhat solved the NL2SQL problem in research after only a few months of work.However, in production, we weren’t breaking the state of the art. The model still made a few mistakes on the user's input. And it became clear to me that changing the model architectures would only get us so far, and that the most crucial thing we could do to improve the performance of NL2SQL was to improve the training data. This was done by generating edge cases synthetically and therefore increasing the training data coverage.So, we started synthetically generating training data (NL-SQL pairs) by following syntax requirements, snowflake schema, etc. However, the data that we have artificially created couldn't possibly cover every possible scenario or edge case. Because the true edge cases are ones that you can't predict or plan for, like the questions that people have typed into our search query bar.&nbsp;Another way to increase the coverage of the training data was to label the user inputs and feed them back to the training data. Taking care of that lifecycle, also called an auto-labeling pipeline, was quite an operation. Even though I was given a lot of resources to manage the auto labeling pipeline, I couldn't undertake any other engineering work because it required so much overhead.So I started looking for the best third-party training data provider in the market, and that's how I met Scale, and the rest is history.<h2>What do you enjoy most about working at Scale?</h2>I like that I get to work on problems that are important and challenging enough to customers that they are willing to pay to have them solved. This means that the problems are real.I also like that Scale's work is always about data, which is my favorite subject. For every business, I believe data is the most valuable and private asset. Say if you're developing a machine learning model, the one and only input that determines how well your model will perform is your data. The fact that customers trust us with their data and allow us to manage it energizes me, and I feel a great deal of commitment and responsibility. Because the data we provide with them is being used as a bloodline for their business, this relationship just can't work without trust. Having a close relationship with customers is also a rewarding part of my job.Another highlight of my time here at Scale is the everyday connection with my team.&nbsp;When the engineering team works well, it feels like a well-oiled machine, and you will notice tangible results from that. We've done some seemingly impossible jobs together, and it feels amazing to see the whole team pulling together to complete a challenging project. When working on projects, you learn a great deal by openly debating and choosing the best technical solution with your team. If you're an engineer, I believe that seeing and interacting with other engineers is one of the best ways to learn and grow. That's also something I enjoy at Scale.<img src="https://lh3.googleusercontent.com/v89cOzzb19dslCKXi3DbqYG0PF8Tjl2n6BxD8hPyUz931p6bQ8v1dIJfgxIoPSVBYe1dlC1uh-GF8_a7BSqreqsA4_lBIuluy66komaVyRGC5IW9Q-j02_KU0eA5JNNZEsBJZLdpt7FCLHV-0GjOXcE"><h6>The team traveled to Cancun to celebrate the project's completion.</h6><h2>What makes working for a startup unique from other jobs?</h2>I've worked at both a big tech company and a startup, and they are indeed quite different, and each one is unique in its own way.At a startup, leading a project feels like driving a sports car. You're close to the ground and can feel the movements you make. You can turn quickly, and you can observe fast motion while you are steering. You can see the results of your team's work fairly quickly and clearly. For example, you complete a pilot project successfully, and the customer signs a large contract. There is also a good chance of crashing - for example, you fail to deliver a project and lose a customer. At a startup, you will see the results of what your team does right away.At a big company, running a project is more like flying a spaceship (I can only imagine this). You’re farther from the ground, it's harder to make quick decisions because you are not the only stakeholder, and the course has been set long in advance. There are also very well-built manuals and processes to guide you along the way.So far, I've enjoyed seeing the immediate, direct effects of my team's work at Scale.&nbsp;<h2>What's your favorite part about being an engineering manager at Scale?</h2>One of the things I like about being an engineering manager is that I get to work on large-scale projects with a strong, cohesive team, which results in a scalable solution and a greater overall team output. Apart from the collaborative and multiplier nature of the job, I also enjoy problem solving at Scale.People think that a software engineer's job is to write code, but I don't think that's a complete picture of an engineer. A more accurate job description would be "a problem solver", since our main goal is to build things that help people solve problems. You won't be able to solve problems if you blindly write code, especially from the wrong angle.I believe the most important thing to do in order to solve problems is to accurately diagnose them. So the ability to diagnose problems accurately is, in my opinion, the most important skill for an engineer to have. If your diagnosis is wrong, you will rarely be effective in solving actual problems. And you can only make the right diagnosis if you have a full, concrete understanding of the problem, the data, and the use cases.&nbsp;While surgeons can rely on other professionals like physicians, x-ray technicians, to aid with diagnosis, ML/SW engineers have to diagnose problems on their own. We have to carefully examine the available data, debug the code thoroughly, then choose the best solution and also implement. The ML/SWE profession comes with a significant amount of both power and responsibility.If you are someone who enjoys doing things like this (proper diagnosis and implementation), you will do well in the field of engineering.<h2>Tell me about interesting projects that you ran at Scale?</h2>Every project at Scale is interesting and unique in its own way, as each project has different data modality (3D Lidar, 2D imagery, NLP, tabular, etc.) I enjoy that I get to work on diverse problems and apply the peculiarities of each data modality to ML model development. For example, 3D data often comes with the new axes - yaw, pitch, roll, unlike 2D data and the sensor's effect on the LIDAR data is significant. NLP also has its own quirks, such as different types of tokenizers, vector representations, etc.These days, my team is working on multi-modal (Language + Vision) problems. For example, given an image of a model wearing clothes, can we describe what she’s wearing in detail (e.g. short sleeve, polka-dots, long boots, etc.) in natural language? To this end, one popular research/exploratory approach is to train a multi-task (e.g., Visual Question Answering, Image Captioning, Text-to-Image Generation, Visual Grounding, etc.) model and use it for attribute prediction.Although I can't talk about the details of the projects I work on for clients, I do make an intentional effort to make my team's work more universally applicable, scientifically relevant, and therefore publishable. The attribute prediction work (link) that was just accepted at CVPR 2022 is a great example. One of the most common requests we get from clients is to assign visual characteristics such as color, shape, or motion to detected objects. But there aren't many researchers working on this topic publicly (though there are a lot of practitioners working on it in industry), and there aren't many benchmark datasets for attribute prediction. So we did in-depth research on this topic, presented our findings at CVPR, and received overwhelmingly positive feedback from the ML community.The Fitzpatrick 17k project with MIT is also a good example. The Fitzpatrick scale is a 6-point scale that tells how dark your skin tone is. And this project’s motives are quite fascinating. Because medical imaging doesn't have much data on darker skin, ML models trained on these datasets have trouble recognizing dermatological illnesses on darker skin.&nbsp;We wanted to bring attention to this problem and encourage more inclusion of images with darker skin tones in datasets. So we annotated 17k clinical images in DermaAmin and Atlas Dermatologico, the two major dermatology atlases, using the Fitzpatrick Scale.&nbsp;We trained a few models on this new dataset and then looked at how well they work on images with different skin tones. And yes, adding more diverse data to the training set was an effective way to reduce bias.&nbsp;Because of this work, researchers and practitioners&nbsp;are now a little more likely to take into account and examine biases in training datasets before developing ML models in the healthcare sector. Ultimately, we want to make sure that ML models trained on imbalanced datasets don't make healthcare inequalities worse by accident. With this study, we received an honorable mention award at the CVPR skin imaging workshop.……..If you are interested in any of the topics that I have said above or in working to solve intriguing ML/SWE challenges, email your resume at <a href="mailto:aerin.kim@scale.com" rel="noopener noreferrer">aerin.kim@scale.com</a>!

Read Aerin's stories about what she does and why she loves working at Scale

Aerin is an engineering manager at Scale AI, leading a Catalog ML team

We've had an incredible group of people join Scale—and we’re still <a href="https://scale.com/careers?office=All&amp;department=All#jobs" rel="noopener noreferrer" target="_blank">hiring</a>! Meet the engineers who are accelerating the development of AI applications:<h3> </h3><h3>Albert Sheu</h3>Albert is a fifteen-year veteran of the tech and startup industry, including having worked as the second engineer at Quora and the Head of Engineering at Patreon. His experience runs up and down the stack, working on distributed systems, developer experience, WebGL interfaces, and all areas in the frontend and backend. At Scale he developed the early 3D labeling pipeline, and has also worked on Scale’s Nucleus and Synthetic Data products. <h3>Jack Wu</h3>Jack Wu, previously was a manager at Niantic where he built the Platform Infrastructure team from scratch and led the team to rebuild the Niantic million QPS, planet scale Augmented Reality platform on an alternate Cloud provider. Prior to transitioning to management, Jack was an early engineer at Uber and LinkedIn where he spent most of time building large volume data storage, data analytics and data security systems and saw both companies grow more than 10X during his tenure.<h3> </h3><h3>Yuting Duan</h3>Yuting worked at Google as a Staff Software Engineer and Engineering Manager and studied at CMU. She worked on some of the most large-scale consumer-facing products: Google Search and Google Maps and founded 2 full-stack teams in Google to help users find food wherever they are, whenever they are. <h3>Shah Xia</h3>Shah has previously worked at Bytedance where she was the Chief of Staff to the CEO. She was part of the early team to incubate the TikTok app, and later directed its global expansion strategy. She is receiving her MBA from Wharton. <h3>Sam Clearman</h3>Sam Clearman received his PhD in Mathematics at Lehigh for his work in algebraic combinatorics.&nbsp;He worked at Google using statistical methods to improve Android battery life before joining Scale in 2018.&nbsp;At Scale, he has worked on pretty much everything, but he has always returned to marketplace management and incentive design.&nbsp;In his spare time, he flies planes (sometimes upside down), and enjoys making games such as 2etris [<a href="https://2etris.com/" rel="noopener noreferrer" target="_blank">https://2etris.com/</a>], Down for a Block [<a href="http://d4ab.com/" rel="noopener noreferrer" target="_blank">http://d4ab.com/</a>], and Up for a Camel [<a href="https://upforacamel.com/" rel="noopener noreferrer" target="_blank">https://upforacamel.com/</a>].&nbsp; <h3>Sasha Harrison</h3>Sasha Harrison received a Bachelor's Degree in Mathematical and Computational Sciences, and a Master's Degree in Computer Science from Stanford University, where her concentration was in Deep Learning. As a student, she conducted research projects related to algorithmic fairness in computer vision and NLP, and was an intern at Google and Microsoft. At Scale, Sasha has worked across new products and ML, in particular, helping to develop scalable batch inference infrastructure as well as deep learning models for application to the content understanding data annotation vertical. <h3>Calvin Huang</h3>Calvin Huang met Alexandr Wang at the US Physics Camp just before starting to study electrical engineering and computer science at MIT.&nbsp;While at MIT, Calvin did research in SMT solving and natural language processing. After graduation, Alex recruited Calvin to join Scale as an early engineer, where he built out a large portion of the infrastructure stack and developer tooling, and helped the core systems scale as Scale did. <h3>Vivek Muppalla</h3>Vivek is an industry veteran who has worked in and lead intraprenual efforts across startups and large companies. His experience is leading full stack engineering and product teams. Most recently he was an early member of Unity’s AI &amp; simulation team, and helped Unity become a leader in the Synthetic data and Robotics simulation space. At Scale, he is focused on building our Simulation based products.

Meet Scale's incredible engineering talent 

Scale's Talented Engineering Team

As ML applications grow in size and complexity, testing is essential to ensure integrity and reliability of the application at hand. Given the non-deterministic nature of machine learning (ML) models, testing and continuous evaluation of ML systems is even more important.&nbsp;
That&rsquo;s why today, we&rsquo;re thrilled to announce Early Access to <a href="https://scale.com/nucleus" target="_blank" rel="noopener">Scale Validate</a>, our testing solution for ML models. The predictive performance of ML components is essential to the overall success and adoption of whole products. Progressively developing a high-performing ML-based system is not trivial, because it typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires continuous iteration and experimentation</a>. During these iterations, engineers need to identify new edge cases and improve model performance on those. With every new model iteration, ML engineers want to be sure that they aren&rsquo;t introducing any performance regressions on mission-critical subsets of data.&nbsp;
Using Scale Validate, you can now define data-centric tests for your ML models to avoid performance regressions during development. This step helps ensure that only the best ML models make it into production.
<a href="https://scaleai.wistia.com/medias/ks6a02aqwq" target="_blank" rel="noopener noreferrer">Video</a>
For ML-based products, model performance should always directly correlate with positive business outcome.&nbsp;The performance of ML components is often crucial to the success and adoption of whole products. Getting to a high-performing ML-based system is not trivial but typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires many iterations</a> and will almost never be done. During these iterations, engineers need to identify new edge cases and improve the model performance on those. With every new model iteration, ML engineers want to be sure that no performance regressions on mission-critical subsets of data are introduced.&nbsp;
While ML model testing is key to fast and robust model iteration, there are a few common issues that we identified with existing testing and evaluation pipelines for ML models:
<h4>Aggregate evaluation metrics miss critical regressions</h4>
A common way of validating ML models is to compute aggregate scores and metrics (e.g. mAP, precision, recall, IoU, &hellip;) across fixed test/validation datasets (commonly understood as model evaluation). Evaluation with aggregate metrics is necessary but not sufficient for robust models because they often miss failure modes that are underrepresented due to data biases. Users won&rsquo;t be able to track regressions with explicit checks on such failure modes that were previously addressed but are failing now.
<h4>Coverage for ML model testing is not well defined</h4>
While we can analyze covered lines of code in software testing, there&rsquo;s no such number which informs about the coverage of tests for ML-models. Lines of code still need to be covered, but more importantly, tests need to cover the operational domain of the model input data. The high dimensionality of typical inputs (e.g. an image has more than a million pixels with 3 channels) does not allow for realistic or efficient input-fuzzing approaches.&nbsp;
Testing pipelines are static and don&rsquo;t evolve with the models
Building ML models is an iterative process. So is testing ML models. Not adapting testing and validation datasets results in vanity metrics which in turn oftentimes overestimate the confidence in the models. Continuously evolving test definitions and adding the most tricky examples is crucial to continuously improve the test coverage in the operational domain. This is the only certain path to confidently deploy robust models to production systems.
Accordingly, we built Scale Validate with the following capabilities:
<h3>Scenario Tests</h3>
Setting up unit and integration tests is the go-to practice to ensure correct behavior of codebases. For ML systems, unit tests are designed to test the correctness of individual model components, which is necessary for a reliable model performance. Scenario tests are designed to test the actual model performance and decision-making capabilities of ML models on mission-critical scenarios in the operational domain. Scale Validate allows users to easily define scenario tests on subsets of data (<a href="https://dashboard.scale.com/nucleus/ut_camxgk2yxqgg05rzc4bg" target="_blank" rel="noopener noreferrer">example</a>), using off-the-shelf or custom evaluation metrics. New models are automatically tested against defined scenario tests. Well-defined scenario tests allow ML engineers to catch regressions before deploying a model to production. When a test fails, users immediately see in which scenario the required performance is not achieved anymore (<a href="https://dashboard.scale.com/nucleus/prj_cabt0xkhtxbg048dvqn0?spoof=select%40scaleapi.com&amp;val_ev_gsp=2165&amp;from_unit_test_id=ut_camxgk2yxqgg05rzc4bg&amp;model_page_display=Evaluations&amp;val_model_gsp=1154" target="_blank" rel="noopener noreferrer">example</a>).
<img src="https://lh6.googleusercontent.com/lyc0KYEFc_1GRv-k5m7BPrtijOlmb4ef2xg6J_oVdWJFimn3rfwTjMkpaYs3aUu5PUJI5VjPExMjvC3tuK3V6LLEuAXPlZDRwoT1wfXaTfmFBeZxuq8ov_3OLitltffWIaU_-Q7RJ2y1XbZefMfzkXI">
<h3>Model Comparison</h3>
Model performance rarely boils down to a single metric. With Scale Validate, it&rsquo;s trivial to compare the performance of new models against baseline models (<a href="https://dashboard.scale.com/nucleus/analyze?analyze_compare_model_id=run_cabp73ckv9wg078pk440&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;q=predictions.count+%3E+0" target="_blank" rel="noopener noreferrer">example</a>). You can use either our standard performance metric tracking across scenario tests or our side-by-side model comparison. All charts are fully interactive and users can start by informing themselves from higher-level metrics, then graduating into learning from item-based analysis.
<img src="https://lh5.googleusercontent.com/Ny6n_eMIjJ1kMGRDpWyp47In_FjhErjXbVUS1ou5VvweGqSo_oDjyd4ORb07DmiCrSBPIbkhxQz-v840Nlaj8DmKzFny66BCVsBh9b1z0gMw23Wgq86jkpasU8QdBzp90S9ER5BQBKAIuPAhodUT8e8">
<h3>Slice Discovery</h3>
Scale Validate enables users to automatically identify coherent and underperforming slices of data that do not yet have test coverage. Users can be sure they are always testing against a representative test set.
<img src="https://lh3.googleusercontent.com/tsjA4raZ7xVSuMM9FZR-gLVSn-pX_r99a1r-2hPCTyub75QNG5J6SSRXf2J5c7WiQtWZIHyEgkS78j-ST7u2qooBIDEXC7xZcGKbo6ojWQmdmhBkkLiFPkF7YS7gDw4j5-uvbhjsUXC67OuuxT451fU">
We built Validate as a developer-friendly solution both for API-first and UI-focused users. The Validate interfaces and endpoints make it as easy as possible to get started and integrated into existing pipelines. Today, Scale Validate is used for the testing pipelines of leading ML teams in areas including autonomous driving, surveillance, and industrial robotics.
As Pickle Robot&rsquo;s CTO, Ariana Eisenstein, explains:
<img src="https://lh6.googleusercontent.com/gb9ujMPiBfQOhkiz36npqFR8kY44KymYOC79ac0AQaL4UAK9H2Nm7w3O5bLE7V0xF6w3tOe2E1eRrl57hDs_FMOaZCm9HbzFutL6UmX_s2mPuJ_psw-Aoub5rNo9jynrD_q0yxbtDG8o-QFEwHEkHxQ">
Getting started with Validate only takes a few steps!&nbsp;
<ol>
<li>Sign up / log into the <a href="https://dashboard.scale.com/corp/login?redirect_url=%2F&amp;clear=true" target="_blank" rel="noopener noreferrer">Scale dashboard</a></li>
<li><a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">Explore Validate with public datasets</a></li>
<li><a href="https://nucleus.scale.com/docs/datasets-in-nucleus#uploading-your-data" target="_blank" rel="noopener noreferrer">Upload your data</a></li>
<li>Upload <a href="https://nucleus.scale.com/docs/model-predictions-in-nucleus" target="_blank" rel="noopener noreferrer">models</a> and <a href="https://nucleus.scale.com/docs/ground-truth-annotations-in-nucleus" target="_blank" rel="noopener noreferrer">annotations</a></li>
<li>Compare your models in our <a href="https://dashboard.scale.com/nucleus/analyze" target="_blank" rel="noopener noreferrer">dashboard</a></li>
<li>Select interesting slices of data and <a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">define tests</a></li>
</ol>
Without a detailed model evaluation and testing pipeline, you&rsquo;ll never know which performance points you&rsquo;re leaving on the table and which performance regressions might sneak into your models. Instead, ensure that your model quality only increases with every model and dataset iteration. Test-driven modeling will ensure that you train a production-grade model faster! If you&rsquo;d like more information about Scale Validate, reach out to us for a detailed demo and discussion <a href="https://scale.com/validate" target="_blank" rel="noopener noreferrer">here</a>.

As ML applications grow in size and complexity, testing is essential to ensure integrity and reliability of the application at hand. Given the non-deterministic nature of machine learning (ML) models, testing and continuous evaluation of ML systems is even more important.&nbsp;That’s why today, we’re thrilled to announce Early Access to <a href="https://scale.com/validate" rel="noopener noreferrer" target="_blank">Scale Validate</a>, our testing solution for ML models. The predictive performance of ML components is essential to the overall success and adoption of whole products. Progressively developing a high-performing ML-based system is not trivial, because it typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" rel="noopener noreferrer" target="_blank">requires continuous iteration and experimentation</a>. During these iterations, engineers need to identify new edge cases and improve model performance on those. With every new model iteration, ML engineers want to be sure that they aren’t introducing any performance regressions on mission-critical subsets of data.&nbsp;Using Scale Validate, you can now define data-centric tests for your ML models to avoid performance regressions during development. This step helps ensure that only the best ML models make it into production.<a href="https://scaleai.wistia.com/medias/ks6a02aqwq" rel="noopener noreferrer" target="_blank">Video</a>For ML-based products, model performance should always directly correlate with positive business outcome.&nbsp;The performance of ML components is often crucial to the success and adoption of whole products. Getting to a high-performing ML-based system is not trivial but typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" rel="noopener noreferrer" target="_blank">requires many iterations</a> and will almost never be done. During these iterations, engineers need to identify new edge cases and improve the model performance on those. With every new model iteration, ML engineers want to be sure that no performance regressions on mission-critical subsets of data are introduced.&nbsp;While ML model testing is key to fast and robust model iteration, there are a few common issues that we identified with existing testing and evaluation pipelines for ML models:<h4>Aggregate evaluation metrics miss critical regressions</h4>A common way of validating ML models is to compute aggregate scores and metrics (e.g. mAP, precision, recall, IoU, …) across fixed test/validation datasets (commonly understood as model evaluation). Evaluation with aggregate metrics is necessary but not sufficient for robust models because they often miss failure modes that are underrepresented due to data biases. Users won’t be able to track regressions with explicit checks on such failure modes that were previously addressed but are failing now.<h4>Coverage for ML model testing is not well defined</h4>While we can analyze covered lines of code in software testing, there’s no such number which informs about the coverage of tests for ML-models. Lines of code still need to be covered, but more importantly, tests need to cover the operational domain of the model input data. The high dimensionality of typical inputs (e.g. an image has more than a million pixels with 3 channels) does not allow for realistic or efficient input-fuzzing approaches.&nbsp;Testing pipelines are static and don’t evolve with the modelsBuilding ML models is an iterative process. So is testing ML models. Not adapting testing and validation datasets results in vanity metrics which in turn oftentimes overestimate the confidence in the models. Continuously evolving test definitions and adding the most tricky examples is crucial to continuously improve the test coverage in the operational domain. This is the only certain path to confidently deploy robust models to production systems.Accordingly, we built Scale Validate with the following capabilities:<h3>Scenario Tests</h3>Setting up unit and integration tests is the go-to practice to ensure correct behavior of codebases. For ML systems, unit tests are designed to test the correctness of individual model components, which is necessary for a reliable model performance. Scenario tests are designed to test the actual model performance and decision-making capabilities of ML models on mission-critical scenarios in the operational domain. Scale Validate allows users to easily define scenario tests on subsets of data (<a href="https://dashboard.scale.com/nucleus/ut_camxgk2yxqgg05rzc4bg" rel="noopener noreferrer" target="_blank">example</a>), using off-the-shelf or custom evaluation metrics. New models are automatically tested against defined scenario tests. Well-defined scenario tests allow ML engineers to catch regressions before deploying a model to production. When a test fails, users immediately see in which scenario the required performance is not achieved anymore (<a href="https://dashboard.scale.com/nucleus/prj_cabt0xkhtxbg048dvqn0?spoof=select%40scaleapi.com&amp;val_ev_gsp=2165&amp;from_unit_test_id=ut_camxgk2yxqgg05rzc4bg&amp;model_page_display=Evaluations&amp;val_model_gsp=1154" rel="noopener noreferrer" target="_blank">example</a>).<img src="https://lh6.googleusercontent.com/lyc0KYEFc_1GRv-k5m7BPrtijOlmb4ef2xg6J_oVdWJFimn3rfwTjMkpaYs3aUu5PUJI5VjPExMjvC3tuK3V6LLEuAXPlZDRwoT1wfXaTfmFBeZxuq8ov_3OLitltffWIaU_-Q7RJ2y1XbZefMfzkXI"><h3>Model Comparison</h3>Model performance rarely boils down to a single metric. With Scale Validate, it’s trivial to compare the performance of new models against baseline models (<a href="https://dashboard.scale.com/nucleus/analyze?analyze_compare_model_id=run_cabp73ckv9wg078pk440&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;q=predictions.count+%3E+0" rel="noopener noreferrer" target="_blank">example</a>). You can use either our standard performance metric tracking across scenario tests or our side-by-side model comparison. All charts are fully interactive and users can start by informing themselves from higher-level metrics, then graduating into learning from item-based analysis.<img src="https://lh5.googleusercontent.com/Ny6n_eMIjJ1kMGRDpWyp47In_FjhErjXbVUS1ou5VvweGqSo_oDjyd4ORb07DmiCrSBPIbkhxQz-v840Nlaj8DmKzFny66BCVsBh9b1z0gMw23Wgq86jkpasU8QdBzp90S9ER5BQBKAIuPAhodUT8e8"><h3>Slice Discovery</h3>Scale Validate enables users to automatically identify coherent and underperforming slices of data that do not yet have test coverage. Users can be sure they are always testing against a representative test set.<img src="https://lh3.googleusercontent.com/tsjA4raZ7xVSuMM9FZR-gLVSn-pX_r99a1r-2hPCTyub75QNG5J6SSRXf2J5c7WiQtWZIHyEgkS78j-ST7u2qooBIDEXC7xZcGKbo6ojWQmdmhBkkLiFPkF7YS7gDw4j5-uvbhjsUXC67OuuxT451fU">We built Validate as a developer-friendly solution both for API-first and UI-focused users. The Validate interfaces and endpoints make it as easy as possible to get started and integrated into existing pipelines. Today, Scale Validate is used for the testing pipelines of leading ML teams in areas including autonomous driving, surveillance, and industrial robotics.As Pickle Robot’s CTO, Ariana Eisenstein, explains:<img src="https://lh6.googleusercontent.com/gb9ujMPiBfQOhkiz36npqFR8kY44KymYOC79ac0AQaL4UAK9H2Nm7w3O5bLE7V0xF6w3tOe2E1eRrl57hDs_FMOaZCm9HbzFutL6UmX_s2mPuJ_psw-Aoub5rNo9jynrD_q0yxbtDG8o-QFEwHEkHxQ">Getting started with Validate only takes a few steps!&nbsp;<ol><li>Sign up / log into the <a href="https://dashboard.scale.com/corp/login?redirect_url=%2F&amp;clear=true" rel="noopener noreferrer" target="_blank">Scale dashboard</a></li><li><a href="https://dashboard.scale.com/nucleus/scenario-tests" rel="noopener noreferrer" target="_blank">Explore Validate with public datasets</a></li><li><a href="https://nucleus.scale.com/docs/datasets-in-nucleus#uploading-your-data" rel="noopener noreferrer" target="_blank">Upload your data</a></li><li>Upload <a href="https://nucleus.scale.com/docs/model-predictions-in-nucleus" rel="noopener noreferrer" target="_blank">models</a> and <a href="https://nucleus.scale.com/docs/ground-truth-annotations-in-nucleus" rel="noopener noreferrer" target="_blank">annotations</a></li><li>Compare your models in our <a href="https://dashboard.scale.com/nucleus/analyze" rel="noopener noreferrer" target="_blank">dashboard</a></li><li>Select interesting slices of data and <a href="https://dashboard.scale.com/nucleus/scenario-tests" rel="noopener noreferrer" target="_blank">define tests</a></li></ol>Without a detailed model evaluation and testing pipeline, you’ll never know which performance points you’re leaving on the table and which performance regressions might sneak into your models. Instead, ensure that your model quality only increases with every model and dataset iteration. Test-driven modeling will ensure that you train a production-grade model faster! If you’d like more information about Scale Validate, reach out to us for a detailed demo and discussion <a href="https://scale.com/validate" rel="noopener noreferrer" target="_blank">here</a>.

Introducing Scale Validate for Model Testing

If code quality is what sets apart good software systems from bad ones, data quality is what distinguishes great AI systems from the ones that will simply never make it to production. While most of the research in machine learning (ML) continues to focus on models, industrial players building AI systems are focusing more and more on the data. The number of companies adopting a <a href="https://datacentricai.org/" rel="noopener noreferrer" target="_blank">data-centric approach to AI</a> development (understood as systematically engineering datasets used for AI) is growing rapidly across industries and levels of company maturity. According to former Tesla Autopilot lead Andrej Karpathy, engineers at Tesla lose sleep over the dual challenges of structuring their datasets effectively and handling the edge cases that might come up.<img src="https://lh4.googleusercontent.com/bGi4IgNBP_6iLqhMvrAwXJFXRzwRHiQt5iMhU1jvUgHC44K-eJFb5uO_tPueneP6pZu9fkxsWnWUsMhITkK6I2acFC52sC3K_Kzhhk0ogXqNPFHfZD5V7HDx1bQhliZIX-UYR_KKBZjNlkTlswb28I0">Graphic adapted from Andrej Karpathy’s slide deck [Train AI 2018]But why is using the right data so central for building out high-performing ML models? The answer lies in unequal data distribution and low-quality ground truth (annotations).<a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Try Scale Nucleus</a><h3>Not all data is created equal</h3>The core hypothesis behind data-centric AI is that <a href="https://scale.com/blog/improve-data-annotation" rel="noopener noreferrer" target="_blank">you can achieve better ML model performance by training your model on higher-quality data</a>. In an ideal world, your model performs equally well in all kinds of situations. For example, in autonomous driving, you would want a model detecting pedestrians to work equally well, irrespective of the weather, visual conditions, how fast the pedestrian is moving, how occluded they are, et cetera. Most likely however, your model will perform much worse on cases that are more rare—for example, a baby stroller swiftly emerging from behind a parked car in inclement weather.&nbsp;The point of failure here is that the model has been trained on data that was recorded during regular traffic conditions. As a result, the representation of these rare scenarios (as a portion of the entire training dataset) is much lower compared to common scenarios. Below is another example of two highway scenarios, whereas lane detection will be significantly more difficult in the right hand picture compared to the left.<img src="https://lh5.googleusercontent.com/Nr-B0CIcVGPiYOS9BRizyV96w9k_tLKcdLpaDpYzoZeTfDB2LOY5F5--BKhH70lyDC9ZBvqxoL2ks2x1C_qIfoO6cBaScI-SkyFTGUCv3GU5NjkT8GpsF788mQpm-bl1yqnVMabnW8umPNOajNgySlE">What can we do about this? We need to increase the prevalence of these rare cases in our labeled training data.<img src="https://lh4.googleusercontent.com/8VXzOph0ZPD3peHbzOCCOgX1Ecx3TbroIKlFlD0-il7MTqFkLVHz7HBwRaYwAINnOMzTZnussN0dS-ivxzk4Flb5VX37a-uAvlXhMuuVKXHBpvakXAkUFkMjQuW_lmxsdpwynICGXNNz_pVCPtR6bX0">Accuracy improvements with data selection via active learning versus random sampling given a fixed labeling budget.Source: <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Emam et. al., 2021. Active Learning at the ImageNet Scale</a>In ML research, this approach is often referred to as active learning. Many papers (e.g. <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Active Learning at ImageNet Scale</a>) have demonstrated the effectiveness of this approach, showing significant performance improvements by picking more difficult (most often underrepresented) training data instead of random sampling. At Scale AI, this is also what we observe with our customers, especially those with advanced ML teams and models deployed in production. Companies that are the best at picking the right data for labeling—and eventually training—have the highest performing models. Unfortunately, identifying the highest impact data is difficult, and most ML practitioners perform this task manually, even when wrangling very large datasets.<h3>Labeling errors are breaking your model</h3>To make things worse, almost all datasets include labeling errors. Labeling errors are more likely to occur on edge cases. This makes sense, because images that have been seen 1,000 times are easier to label correctly compared to an image seen only once. Labeling errors are also most destructive for your model performance if they happen on edge cases, because you have a limited amount of these samples in the first place. So if the labels on these difficult samples are wrong, model performance will never improve. Instead, it’s more likely to quickly regress.<img src="https://lh4.googleusercontent.com/rFWziHErcB0S1tlTzQQDw2OWyhFgaCHQJumi-rYeMg0Wtmql4_Xr31QXo8bZrBo_xt_Nhk5_XflX7BEW4cOaJmoqS0bVcrPviFCzeKQC8wEAQbytWbb-FsJghbtm0tnqREVjq_NJdoplyM3u6_UEFxs">Example of a typical l labeling error on the public Berkeley Deep Drive dataset: class confusion between car (the annotation)&nbsp;and bus (the prediction).&nbsp;As a result, finding and cleaning bad labels (generally, but especially for rare cases) is probably one of the most impactful activities you can do to meaningfully improve model performance. Many teams do this already. Unfortunately, though, doing this consistently, especially on large datasets while keeping track of issues and measuring progress, is often a very manual and isolated process. Many times, teams will use Jupyter notebooks to find bad labels and keep track of them in spreadsheets.Doing things manually will only work for so long, and it will not get your model into production anytime soon. To address this we have built Scale Nucleus, a data management tool that helps to decide what data to label and make sure that these labels are consistently of high quality.In the rest of this article, we’ll walk through common strategies to mitigate the problems outlined above, starting with unequal data distribution.<h3>Use similarity search to get from one failure case to many</h3>One of the most common ways to build good training data is to look at a single sample that is particularly difficult for the model (e.g. based on pure luck, a field report, domain knowledge or other data points) and try to find more of the same.<img src="https://lh3.googleusercontent.com/JKZZC8hjToji6ebsTT9Tm6iAk-nY3gX-BPs4j-Ojc76bp_J3BXcT-kU1vZJ9njE2f0KYt3opfz9n8cQW2bf5_DMRB40XtXkvBfRbkw7P6Y-tVpKTCU4mz7-Z9jFFh5soMJ6t79KOoyxZDMzojbi_FRk">In Nucleus, you can do this by selecting a few images of interest in your dataset and running a similarity search. For example, you might know that perception performance is bad on blurry images. In this case, you’d select a single blurry image and run a similarity search.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F3c1101b17e1d4d95a3030dec3104f835">However, a single similarity search might yield mixed results; not everything is equally relevant. Furthermore, you might want to codify the issue you were looking for into a “tag” and consistently find all images that have this issue. In our example, you might want to make sure every image in your dataset can be identified as blurry or not blurry going forward.One way to address this is using a Nucleus Autotag. Autotag is a way of teaching Nucleus about a new concept, by interactively selecting more and more specific examples of a case, for example blurry images, in a few refinement steps. In just a few minutes, the platform will “learn” a concept like blurriness and enable you to search your data using this attribute going forward. In the background, we are training a binary classifier, which assigns and updates a similarity score for the concept you are searching for across your whole dataset. You can learn <a href="https://nucleus.scale.com/docs/autotag" rel="noopener noreferrer" target="_blank">more about Autotag here</a>.<a href="https://scaleai.wistia.com/medias/gd6yfivwwk" rel="noopener noreferrer" target="_blank">Here</a><h3>Find rare edge cases with natural language search</h3>Many times, an ML engineer might know (e.g. based on domain expertise) what kind of edge case data is difficult for their model, but does not immediately have an example of this at hand. Edge cases are often also environment conditions or situations, for which you have neither annotations and nor does your model predict them—yet. Examples might be specific lighting conditions, occlusions, or novel objects types.&nbsp;<img src="https://lh6.googleusercontent.com/ZVoRKE_yYTYVkKMTLeXN1vuh-3ExukAYnhTIPL9vtW55Bee0ePoKU5tycAWXJwytidezHuu_f5jvbInP5-P2GLYlqUPmTS-WM0ocfqEVwfJr-G1lNSmUIIHtPhwYnu3f4ke3tGXqoutxCCkXzBOhPmw">To start, you might run a Natural Language Query to find these examples. For example, you might want to find <a href="https://dashboard.scale.com/nucleus/ds_bwk18y3e696g05rk8gyg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+pizza+box" rel="noopener noreferrer" target="_blank">all frames of pizza boxes</a> in the famous ImageNet dataset. It is important to note that this style of query works on data without any annotations, model predictions or metadata because the samples you are trying to find are edge cases which are not yet detected by your computer vision model.One way of solving the “no labels challenge” is to use precomputed embeddings that are matched against a natural language query, e.g. using OpenAI’s <a href="https://github.com/openai/CLIP" rel="noopener noreferrer" target="_blank">CLIP</a> model. Nucleus generates CLIP embeddings automatically for all images in order to support natural language search.<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">Natural Language Search</a>Once you’ve built a more balanced dataset and enriched it with annotations, now you’re ready to ensure these labels are consistently of high quality, especially for edge cases.<h3>Review computer vision labels</h3>Now that you have a set of image annotations, you’ll want to verify that they are of high quality before using them as training data, especially for rare edge cases. This can either be done image by image, or in batches as long as the data annotations can be easily inspected and compared against the actual image content.<img src="https://lh3.googleusercontent.com/AdVeuzXqgQzQKvjjMaP4j2Dco5iGD7CLrs2FXOriHzr0I48c4J732dkeZ6Ynx_TAPlculOjwyWnWKB0HinRuzaXG--Ays5RULi1ptcMF8DmPqxKZodGnFc-gDnx98xNsZjfyzFM995-yG2wMUZkPcWM"><img src="https://lh4.googleusercontent.com/GYD7xBOcas6YKhtCKoHZWwxib2KGMRJXXRN-X_hwGzek_UO37C_gQiatduhSZKO5sKXPd09li3pDYA64kMqeN3viU8IjhYCtMq5AkNsbK-xOg7AnN5gd9KSdQqy0XF5EhGtoN_yCrxt8gOl-QYRtUwA">To improve label quality in the next iteration of labeling, it makes sense to mark each individual frame and bounding box as accepted or rejected and later query against these attributes to make sure all items are reviewed. If you encounter a particular issue, such as a badly fitted bounding box, a good option is to add the relevant images to a “slice”, which is a curated subset of data to share with colleagues or to return to at a later point.<img src="https://lh4.googleusercontent.com/UtXwA46VnC7onh-V4MrgPgF8cFHJZ3wxDeL4NJEb8PGoX8atpZsRSRMR9uIN29g5W4uVrjGxHve6mZHk734ZBAQpvY86X-nE9JaKN832GKyY3d4_oUubJR4L-DkVF5997tBqog57l1aNYKDfPTnAu7g"><h3>Find mistakes in computer vision labels</h3>Instead of going through all your data manually, it is often advisable to compare the labels you are reviewing against the predictions of your most recent computer vision model. There are several so-called model-assisted methods to find and clean labels such as sort-by-loss, sort-by-confidence where model and ground truth disagree, or confident learning (<a href="https://jair.org/index.php/jair/article/view/12125" rel="noopener noreferrer" target="_blank">paper</a>). It is in fact often surprising how effective methods such as sort-by-confidence are to improve label quality. ML practitioners do this by computing common error metrics such as the aggregated IOU, a confusion matrix, or a precision-recall (PR) curve (e.g. in a Jupyter notebook) and then inspecting the samples where the prediction of the model and the data annotation are least aligned.<img src="https://lh3.googleusercontent.com/me0p6swzqwP1_AN4g8O4BbGBuKNb0LqJLgqr240I39VTN2a-jaHLYSC-VUn5fJAFkhf6z9ClYSg9OKZs5TC-UEJzEA1tzg5IwMoiH-2Auf0ECZOJhUwxT-uIIhgGh0TNaWfkXHW0KsI2A6hrv7D8vmo">However, using a GUI this process can be 10x faster, because error metrics are pre-computed automatically and can be directly explored with interactive charts. Clicking into any graph automatically queries and surfaces all the samples associated with that data point. For example, clicking into the cell for “bus versus truck” confusions in the matrix displayed below queries your dataset for all samples where the data label is a truck but the computer vision model predicts a bus.<img src="https://lh4.googleusercontent.com/9GiOAPH8IoxI6QiMfYa7Ia7rOPcvwDTZuuMaDkPTG7-176VwjOM5cjZmLUGNXEBvwtpCclB3Ryp3e8vQV6BHMpEagh14TbEc_L90i7vuKb7KX04Xkvxt_Pjap7Ta7LShMBKcRzYfVDG35ecCdaEdjJE"><img src="https://lh4.googleusercontent.com/XFAtHxplLOV_1YHFQJv6Bb80wU2qce2hu-20b-D-BzkIIJ0n8xl6iNAgdxMuUse3ckJdEcbGUEEMzC1qhyNAdzls6lFQiU8P6JkPLZd1KxuRY_Iwuy6zS_ZQZMvVvtfMUZqeQX-6RI-A6-sbYMWVXrE">From just a single glance, we can see that these are buses that are incorrectly labeled as trucks. These image annotations need to be corrected to avoid bad model performance for both bus and truck detections.<h3>Find missing labels</h3>Finally, it is critical to ensure that no labels are missing in your annotated image data. A missing label can lead to a performance regression of your model, especially if the object with the missing label is easily detected by your model.<img src="https://lh4.googleusercontent.com/IlPV_qOBq4bmtcTAEP119G257XAKc3W5titad6vm2VF0rZ25lIGQhALm2PcLwlB1QUS8K3bTKf17Lcqf24zed-f-mPhLtPEJqk9V9m-eF6Bnz6D7ueyoeSqs10ra1YO_PHqXwFmtf9LO30pJbSQul2Y">In the above example, all cars are annotated in the image, but the single white vehicle is just clearly omitted from any detections. A perception model for vehicles will easily detect this car. However, if it is re-trained with the shown bounding boxes as ground truth, it learns to disregard white cars, which would likely be a dangerous failure mode.<img src="https://lh5.googleusercontent.com/4B7S6d1FE4cK_UP14Ma9iDx2wd7Rq2M1yHUInc7UqZKW6VXOoNUFmr_FNlsZFTiwSPlBPa-k21oGgGEegKF6AM3X5X5QZcjzpT__NNMMhLXw71IWjNrhIT8kJamypgptKEC2308JFeNFuUYpRkOi9P4">The confusion matrix is again a good starting point to find these cases and make sure they are corrected. We can find the row with “no ground truth” in the bottom and query for examples where the model is currently still predicting an object.<img src="https://lh4.googleusercontent.com/F5tkfSRWH0rvhyDN4okzsTBgsse8qDz9KwiNe6FD-5IFbDgactKLJyEPDTZpllDriNkf4ghppUwDV6rhkU826ijXHLRQqaKsxwJ7pFY4BY2Hkda5VN2mwRJC8Ko4Ow0s0uyA15jENUHUwK-xEC2kkfE"><img src="https://lh3.googleusercontent.com/vexNOiYxXuY_0_P8Ani_ao266gUZbJFRaYxSNsuisQbZbYVgbRzlP93fXzdB7J9jREzqYnGdBAEXZ9xWaoZcNdiKfxFf9AMDIXYks2TDL1e_8FgzlBh8YXTOsqeylMdR2CgrhhQB6VcTHWjlshkNmNY">With this query, we can swiftly find all cases that are obvious misses and make sure that the data annotations are corrected for these images.<h3>Dealing with long tail data distributions: the TL;DR</h3>We talked about two common problems when training high-performance computer vision models: unequally distributed data with rare edge cases and bad image annotations.Both research and industry agree that picking out the right data to annotate leads to much better model performance, because it corrects for the naturally unequal data distribution encountered in the real world. The model can learn how to deal with edge cases by being trained on more data that contains them. Secondly, almost all datasets contain annotation errors, while computer vision models work on the assumption that all annotations are correct. This causes the models to perform worse, in practice. This problem is most pronounced for rare edge cases, as these are harder to annotate and they have a worse impact if they are annotated incorrectly.There are proven methods to cope with these problems. Good strategies to pick high quality data include starting from difficult images and finding more of the same or running semantic search for situations that are known to be problematic. Second, we have learned that reviewing the quality of data annotations is essential. Using error metrics that compare annotations with current model predictions (e.g. with a confusion matrix) is a good way to quickly surface the lowest quality data labels or even missing annotations.Getting the edge cases right is considered to be the “holy grail” of building high performance computer vision models, and with the right techniques, you don’t have to lose sleep over them any longer.<h3>Ready to try Nucleus yourself?</h3> <ul><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+woman+with+a+baby+stroller" rel="noopener noreferrer" target="_blank">Find edge cases from scratch with Natural Language Search</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=image&amp;display=similarity_image" rel="noopener noreferrer" target="_blank">Find similar images from a single sample</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=bus&amp;order=Desc&amp;metric=nucleus_confidence&amp;annotation_labels=truck" rel="noopener noreferrer" target="_blank">Find mistakes in computer vision labels (i.e. class confusions)</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;explorer_patch_display=false_positive&amp;iou_extent=0%2C0&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=car&amp;order=Desc&amp;metric=nucleus_confidence" rel="noopener noreferrer" target="_blank">Find missing image annotations</a></li></ul>

 A guide for ML engineers losing sleep over edge cases

Don't lose sleep over long tail data distributions.

How to Handle Long Tail Data Distributions in Machine Learning

Within the field of computer vision, semantic segmentation is a close approximation to describing the visual world the way humans perceive it: entities with precise boundaries. For use cases where nuanced scene understanding is mission-critical, like autonomous vehicles and intelligent robots, building models requires massive amounts of highly accurate segmentation masks with pixel-level precision.The exactness required near object edges that makes semantic segmentation valuable also makes it incredibly time-consuming for humans to annotate.&nbsp;Given taskers annotate thousands of images per day, we wanted to accelerate the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotation</a> process without losing fine-grain precision. To accomplish this, we developed a feature called Autosegment. An annotator draws a box around an object and a segmentation mask is automatically generated. The annotator can then adjust the mask before marking it as complete. Under the hood, the subset of the image is sent through a model, which automatically identifies and segments the object of interest.<a href="https://scaleai.wistia.com/medias/lvnzhzfl7b" rel="noopener noreferrer" target="_blank">video</a>The Autosegment functionality was originally developed for autonomous vehicle data, which is a core customer use case at Scale. (You can read more about our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">original Autosegment tool</a> in our previous blog post.) When it was first launched, we saw a 30% decrease in the time it took to produce segmentation masks. However, the use cases for Autosegment have expanded greatly since its introduction, with AV data now constituting only one of many domains for which Scale produces segmentation data. While the underlying model did generalize beyond the autonomous vehicle domain, we observed that it performed noticeably better on objects like street signs, vehicles, and pedestrians than on other types of classes.This article contains a technical deep dive into the machine learning model that powers Autosegment. We’ll walk through some of the choices we made during the development process as well as our recent effort to overhaul the model for improved generalizability. In particular, we will discuss the data-centric approach we took to boost the usefulness of Autosegment across a wider range of use cases.<h2>Model Anatomy</h2>The segmentation model developed to power this tool addresses a problem known as salient object detection wherein the goal is to identify and segment the most “salient” or prominent object in an image. There is subjectivity in this task because an image can easily contain more than one prominent item. The intention behind Autosegment is that the user draws an approximate bounding box around the object of interest. Based on this we can firmly make two assumptions:<ol><li>The target object will be relatively centered within the box</li><li>The target object will extend nearly to the edges of the box, with some nonzero padding</li></ol>We encode these assumptions into the training process. Specifically, images in the training set are cropped in such a way that the target object is always centered with some randomly added padding. For any training image containing multiple segmented objects, each object is cropped separately to produce an individual training observation. <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fe23c12266e384e7ca60a35f93258ba3e">The architecture we chose to implement for this purpose is based on U2-Net proposed by <a href="https://arxiv.org/pdf/2005.09007.pdf" rel="noopener noreferrer" target="_blank">Qin et al.</a> for this type of problem. U2-Net provides two concrete benefits over other architectures we considered.First, it is designed to more effectively take into account both local and global contrast features of the image. Most of the popular feature extractors (VGG, ResNet, etc.) were designed and optimized for image classification. While they can work well for segmentation tasks, they do so by identifying features specific to the semantic classes they are trying to predict. The consequence of this is that they tend not to generalize as well to classes absent in the training set. On the other hand, U2-Net is designed to make use of image features that indicate object saliency. Because of this, while U2-Net may not perform as well at outright classification as more traditional architectures (we did not test this), in our testing it does a good job at identifying salient objects pertaining to obscure classes. This is important for Scale’s customers who often need to annotate data for niche use cases not represented in the training data.Second, we want to carefully avoid the need to stretch or downsample images. Because the goal is pixel-perfect accuracy, the model needs to output a high-resolution segmentation mask. Downsampling, stretching, or otherwise modifying the image in any information-reducing manner before feeding it to the model will typically result in blurrier segmentation masks than desired. To this end, utilizing a fully convolutional network allows for the processing of any image crop size or shape as long as it is large enough to be processed by the encoder phase of the U-Net (and small enough to fit within GPU memory). In addition, unlike some other architectures, U2-Net does not excessively downsample the input with convolution strides or pooling. This helps propagate enough information about the original image through the network to produce high-resolution mask outputs.<h2>Improving Generalizability</h2>Neural networks have a tendency to try and “cheat” by taking shortcuts when learning to solve a problem. Faced with the task of salient object detection, one shortcut is to simply learn how to segment the specific classes present in the training data. For example, if trained exclusively on autonomous vehicle camera data, the Autosegment model only needs to memorize what certain object classes look like, such as cars, trucks, and pedestrians. The resulting model may work for annotating vehicle scenes, but it will not perform well when applied to new object classes because it hasn’t truly learned the concept of “saliency.”<a href="https://scaleai.wistia.com/medias/dhzm8kaohk" rel="noopener noreferrer" target="_blank">Video</a>To overcome this challenge, the model needs to be given training examples for which this shortcut does not work. In the case of Autosegment, we expanded the training dataset to include hundreds of new object types. Neural networks have a finite number of parameters,&nbsp;meaning that there is a limit to how much “knowledge” they can hold. By providing more classes than the network can effectively memorize, the hope is that it will be forced to learn a simpler strategy that can detect saliency across a more general set of images.Our first step towards generalization was to incorporate the open-source <a href="https://cocodataset.org/#home" rel="noopener noreferrer" target="_blank">Common Objects In Context (COCO)</a> segmentation dataset into the training process. However, while COCO contains a wide variety of object classes, it only contains about 200 thousand annotated images, which is a small fraction of the training data from autonomous vehicle cameras. In addition, COCO segmentation masks are often somewhat imprecise.We then sought to leverage the vast segmentation data that has been produced using Rapid, our annotation platform for rapid experimentation. Rapid supports both full-scene segmentation and partial-scene segmentation. In full-scene semantic segmentation, every pixel in the image is classified into one of multiple classes. In partial-scene segmentation, pixels may be categorized into zero or more classes. Hundreds of thousands of images have been annotated using Rapid.<a href="https://scaleai.wistia.com/medias/15nwn1o6ff" rel="noopener noreferrer" target="_blank">Video</a>Full-scene segmentation often contains classes that are not beneficial to train on. For example, in outdoor scenes the sky is often a distinct class in the segmentation mask but is typically considered less salient than all other objects in the image. While we want to train on a diverse set of object classes, we do not want to train on “background classes” or objects that do not match a reasonable definition of saliency. Neural networks are generally robust to occasional adverse training samples, but in large quantities they can negatively affect training. In order to filter out these undesirable classes, we have taken the time to construct a mapping between a common taxonomy and the customer-specific classes. For example, this mapping helps us recognize that what one customer calls a “bicycle” is the same as what another customer calls a “bike.” From this, we filter out segmentations that do not match a list of approved classes. Partial-scene segmentation is the more common project type on Rapid and mostly avoids this problem of extraneous classes.<h2>Validation</h2>An important consideration for us was that because Autosegment is already in use across a large number of projects and use cases, changing the underlying model risks unknowingly worsening its accuracy on specific categories of images. In theory this type of accuracy loss could occur due to the finite information capacity of the model architecture. As mentioned earlier, the network needs to encode its problem-solving process using a limited set of parameters. If the number of parameters is too small, underfitting will occur and impede accuracy. We combat this possibility in two ways.First, we ensure that the training data is well-balanced. If the overall composition of the dataset is skewed in favor of any particular domain, then the model may tend to overemphasize training samples from that domain. Not only can this result in lower accuracy in other domains, but it could impede our goal of forcing the model to learn a generalized solution to the salient object detection problem.Second, in addition to benchmarking accuracy on the overall collection of images, we also benchmark the model’s performance on subdomains (such as autonomous vehicle data) to ensure it performs well within each of the strata. This essentially serves as “regression testing” for the model. Regression testing is a software development practice to ensure that code changes do not break existing known use cases. To accomplish this, we validate the model on independent test sets representing distinct customer use cases. The crucial use case we monitored during these improvements was autonomous vehicle data:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fdb0470f15fa945cda7b946e690837b34">Beyond standard pixel classification metrics, we also calculate the precision and recall of pixels within a small neighborhood of the segmentation edges. These metrics help us assess how well this tooling will actually benefit annotators for whom the most time-consuming aspect of semantic segmentation is the object boundary pixels.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fac64ee3d7a814bc0852260ed4e5721f9">To calculate these edge metrics, we first start with the predicted and ground truth masks. Keeping only the single-pixel boundary of each mask, a <a href="https://en.wikipedia.org/wiki/Dilation_(morphology)" rel="noopener noreferrer" target="_blank">dilation operation</a> is applied with some small threshold. Metrics are then calculated from the dilated edges. The above image visualizes this process.<h2>The feedback loop</h2>Beyond the initial training of the model, data annotated with Scale Rapid continues to get fed back into the training process. This creates a feedback loop: Autosegment helps users to more quickly produce higher quality annotations, and in turn, those annotations serve to continuously improve the automatic segmentation accuracy. Having high-quality training annotations is an indispensable prerequisite to training a high-performing model. This <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> lifecycle is a unique characteristic of Scale’s annotation platform and fundamental to how we achieve best-in-class annotation speed and accuracy.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F03a3cce06e0e4cfba0dd090356f2669d"><h2>How you can use Autosegment</h2>Autosegment is one of the multiple Active Tooling capabilities that exist in Scale’s products. Active Tools provide interactive ML-powered assistance to data annotators. By letting humans make changes to the model inputs in real time, model outputs can be quickly dialed in to match the situation precisely and minimize time spent on the task.You can supercharge your data annotation using <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a>, which offers best-in-class annotation infrastructure to accelerate your in-house annotation team. Studio also offers a free tier to familiarize yourself with the platform and start annotating data. If you’ve decided you’d like to offload more parts of the data annotation pipeline, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> provides an easy way to send your data to be annotated by Scale and get production-quality labels with no minimums.

Engineering

How Scale AI’s computer vision annotation tooling (CVAT) uses machine learning-assisted tooling for faster and more accurate semantic segmentation.

Machine Learning-Assisted Image Semantic Segmentation

Machine Learning Assisted Image Segmentation

<h2>About me:&nbsp;</h2>My name is Sophia Luo and I am currently an engineering manager and product manager on Scale Rapid, our self-serve and rapid experimentation data annotation product. During my time at Scale, I have been an engineer and product manager on several different teams; including eCommerce, Marketplace, and Platform. In four years, I graduated with a Masters of Engineering in Computer Science, Bachelors of Sciences in Computer Science, and Bachelors of Sciences in Mathematical Economics from the Massachusetts Institute of Technology. In my free time, I like to dance, draw, eat good food, and plan fun and new experiences with friends.&nbsp; <h2>Top Slacker at Scale AI&nbsp;</h2>When I first graduated college, I kept telling my friends how confused I was trying to decide where to go for my first job as a software engineer.&nbsp;Fortunately, I had the opportunity to choose a path that was best suited for me, as I intentionally reviewed multiple offers. This was a major life decision, the biggest one I had to make since deciding which college to attend four years prior.&nbsp;When making this decision, I accumulated 18 different company traits, including growth opportunity, work life balance, mentorship, and more that I would evaluate to see if the position would be a good fit. In hindsight, the whole process was very heavy handed. In the end, I realized that there were only a few traits that really mattered to me and it became clear that Scale was the best choice:<ol><li>Growth opportunity: Scale was quickly growing, and I knew that there would be many organic career opportunities. It also became apparent that the company was open to providing opportunities on important projects to those willing to take them on– regardless of tenure or experience. This was very important to me as someone starting their career. I wanted to be able to dive straight into the deep end and learn by doing.</li><li>Talent: The high concentration of talent at Scale was awe-inspiring. Just from talking to the employees and searching their profiles on LinkedIn, I was both intimidated and excited to join a group of people who were so talented. I wanted to surround myself with and learn from some of the most talented people in the industry.&nbsp;</li><li>Culture: After spending time with Scale employees and also hearing about their experiences, I found that there was a very tight-knit community of friendship and trust. As a fast growing startup, there were ups and inevitable downs, but it was clear that the Scale people had a very tight bond that helped them get through even the toughest of times.&nbsp;&nbsp;</li></ol>Fast forward about half a year and — boom — the pandemic and quarantine. The last semester of my college career was cut abruptly short, and I found myself back at home feeling sad, bored, and lonely. As a lonely new grad starting at a company where I couldn’t see the faces of my coworkers by default, I regularly sent cold messages to people across the company on Slack and asked to schedule 1:1 meetings to get to know them.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa8d872b8d3a54833b1932f565b02172e">As the number of people I became friendly acquaintances with grew, so did the number of Slack messages I sent. In fact, I was sending Slack messages outside of my normal work duties so much that I found myself added to our #top-slack-users channel after a few months of starting at Scale.As I was getting to know people through these Slack DMs and 1:1s, I also found myself working with many of them through my first large-scale project. Within my first six months, I was tasked with developing Scale’s first marketplace model as the third engineer on our Marketplace team. Developing this model from 0 to 1 involved applying many econometrics and linear programming concepts I learned in school. I needed to compute annotation pipeline capacity requirements and derive other useful variables. Furthermore, I was learning about data orchestration technologies and systems design techniques to be able to compute across all live annotation pipelines. The implementation and rollout of the model also involved more cross-functional work than I could ever imagine; I was interfacing with product managers, program managers, operators, engagement managers, and engineers across the organization who all wanted to learn about and integrate with my work.&nbsp;For being so early in my career, I felt blessed with the opportunity: there was just so much to learn and I was excited to be owning such a large surface area of work so early on in my career. I felt like I was learning and growing very quickly. In addition to improving my raw technical ability, I was learning how to become a more effective engineer by learning how to program manage, communicate with stakeholders, prioritize feature requests, and drive outcomes.&nbsp;As I was working on the marketplace model, I also became heavily involved in our university recruiting program. I was very excited to write, test, and calibrate new engineering interview questions, and it was Scale’s first time courting candidates completely virtually. Instead of hosting in-person events for candidates to learn more about Scale and the culture, we had to be more creative. I ended up organizing and playing countless virtual Among Us games with candidates, and we also threw together a Trivia Night where we invited candidates and Scale employees to compete for fun prizes together.&nbsp;After quite the eventful first year at Scale, I found myself with yet another career growth opportunity. I had transitioned from the Marketplace team to become the third engineer on the eCommerce engineering team where we were building a new product from 0 to 1 to develop AI-enabled data infrastructure for eCommerce.&nbsp;Being on the eCommerce team was quite the rollercoaster ride. As a team of three engineers, we first worked on a proof of concept for a customer who is still one of our largest today. This was a two week sprint where we built everything from scratch. With the correct licensing and legal practices, we needed to collect raw catalog data, extract and transform the data to standardize information across all catalogs, enrich the data with human-in-the-loop annotation pipelines, and build quality assurance tools and processes in order to ensure high quality delivery of data.&nbsp;Given the specification and deadline, it did not seem possible to meet the customer goals. But, after some long nights of sprinting, problem solving, and teamwork, we exceeded our own and the customer’s expectations – so much so that the proof of concept exploded into a very high volume contract. The two week sprint then became a two month marathon to meet the next customer deadline. We had to productionize all our scrappy processes from the proof of concept in order to scale to more than 1000x the volumes.&nbsp;The team grew from three engineers to around twenty in a short amount of time. I found myself not only executing on my own engineering work but also onboarding and mentoring countless others. Although my title was still officially “software engineer,” I was working on building out parts of the eCommerce product as a product manager and also became the acting engineering manager for some of the onboarded engineers. I was rapidly growing into more leadership-like roles within the team; I owned large surface areas within the product, and led pods of engineers to drive outcomes within those areas. Later, I ultimately stepped up to take on the responsibility of becoming the single threaded owner who led and owned the success of the entire production contract, which involved leading not only all the engineers on the team but also all cross-functional stakeholders.&nbsp;After my work on the eCommerce team, I transitioned to becoming the first product manager and engineering manager on Rapid, one of Scale’s fastest growing self-serve products after having been launched only one year prior. Now, I spearhead numerous workstreams to push us towards the overall goal of being the default annotation platform for all machine learning and data use cases. Every day, I find myself hopping from thread to thread across engineering, product, design, finance, marketing, and sales. I also still code from time to time to increase engineering bandwidth on the team so that we can push new features for our customers and patch inevitable bugs as they come up.&nbsp;&nbsp;The challenges on Rapid are unlike those that I have previously experienced at Scale. Instead of building for a specific annotation use case, we need to build generalizable and scalable product solutions in order to support all data types and annotation use cases. This involves being able to support images, videos, text, audio, and other types of data. We also need to develop the taxonomy and tooling to support all associated annotation types, including classification, transcription, semantic segmentation, and geometries.&nbsp;&nbsp;Given that customers want high quality data as quickly as possible, we also optimize our work to redefine the data quality and turnaround time frontier. To ensure high quality data annotation, we innovate in areas, such as generalizable worker training infrastructure, and ML-assisted validators to flag annotation issues and edge cases. To make sure the data is annotated as quickly as possible without detracting from quality, we also invest in better tooling, ML-assisted annotation (e.g. <a href="https://scale.com/blog/automatic-segmentation" rel="noopener noreferrer" target="_blank">auto semantic segmentation</a>), and pipeline improvements to break one larger annotation task into multiple smaller ones so that multiple people can work at the same time (e.g. our <a href="https://scale.com/docs/video-annotation-with-rapid" rel="noopener noreferrer" target="_blank">video stitching pipeline</a>).&nbsp;While driving all these product improvements and leading some parts of the technical solution, I also partner with our marketing, sales, and finance teams. I experiment with different growth hacking initiatives, such as hosting our <a href="https://scale.com/kaggle" rel="noopener noreferrer" target="_blank">first Kaggle competition</a> and <a href="https://scale.com/rapid/university" rel="noopener noreferrer" target="_blank">partnering with universities</a>. Furthermore, I work closely with our sales team to make sure that we are growing existing accounts and customer segments as well as targeting untapped market opportunities. For every sale opportunity, I am also looped in on pricing decisions in order to make sure that we drive revenue growth while balancing our costs and margins.&nbsp;&nbsp;From building upon my raw technical ability to building a diverse suite of skills as a product and engineering manager, I am grateful everyday for the opportunities that Scale has given me to build my career. I will never forget the memories I have made and the skills that I have grown on the Marketplace, eCommerce, and Rapid teams as well as through my heavy involvement in university recruiting and event planning.&nbsp;The tight-knit community is a constant reminder that we can accomplish great things together while making lifelong memories along the way. Recently, I was somehow able to convince our company leadership (including our very own CEO!) to learn a dance that I taught them for at most thirty minutes. Then, they performed this dance together in front of the entire company during our first company-wide in-person celebration after quarantine.&nbsp;Scale is a great community with some of the most hardworking, talented, and fun people in the Silicon Valley tech scene. I will always be thankful for the highly talented group of people who were with me through the ups and downs. These past two years have been quite the rollercoaster ride, but my reign as the top Slacker of Scale has been quite consistent.&nbsp;

How a new grad grew her career at Scale

Large-scale datasets are at the center of model development and evaluation. Quality of training data is the dominant factor for model performance. Anyone can create an excellent model based on great data, but even the best ML engineers can’t design good models with bad data.&nbsp;If you’ve ever used popular public datasets such as ImageNet or MNIST, you would assume that the class and attribute <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">labels</a> are correct. Shockingly, ImageNet, the most widely used dataset for popular image recognition systems, has a 6% error rate[6]. When models are fed bad data, such as incorrect classes, incorrect features, or missing <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a>, they often have severe classification problems, which can destabilize ML benchmarks and contribute to representational harms in datasets.&nbsp;Examples of representational harms[1] in datasets include under-representation of darker skinned subjects compared with lighter-skinned subjects within prominent facial analysis datasets[2,3], and millions of images of people in ImageNet labeled with offensive categories, including racial slurs and derogatory phrases[5].&nbsp;Even in leading applied ML research groups, approaches for characterizing and fixing labeling errors in massive datasets are limited.&nbsp;One approach to combat noise in a dataset is Confident Learning (CL), which is based on pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. However, CL entirely ignores the case where one or more true labels are missing. This limitation occurs because CL relies on building a pairwise class-conditional distribution between the given and true labels[6]. Other techniques include sorting by model loss, which surfaces the examples that are most ambiguous and difficult to learn, and sorting by confidence where model and ground truth disagree (quite effective!).What is the most principled option after amassing a collection of bad labels? In practice, it would be fixing your dataset by going through the images and fixing the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a> one by one.&nbsp;This is both laborious and impractical.That’s why (1.), a way to visualize, triage, and organize your data, and (2.) a way to fix your annotations, are critical to your data annotation project.Scale provides an integrated dataset and workforce management platform that solves these problems.Let’s walk through an example of how you can use two of Scale’s products: Scale Nucleus and Scale Studio to identify and fix bad labels leveraging model predictions. For this example, we’ll be using the public <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a>.&nbsp;<img src="https://lh3.googleusercontent.com/hYW3_C_Yz4vuaZKQppVBRJB273v9Nz2b9bgHBDbl-WUrdKjfHg31Ic6E2j34WJZSuXSufVkBgfWf0Q7ThfnZ-l7CxScpOViGECQXXHBhk3_VkMqYT0t-GpgGL_1J_O97Vg7wmCjTPh-PSodX-Zm7o-c">Starting on the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=insights&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg" rel="noopener noreferrer" target="_blank">“Charts” page in Scale Nucleus</a>, a simple click into the confusion matrix shows 2,277 cases in which models predicted “person” but the ground truth label is “car”.<img src="https://lh5.googleusercontent.com/e-ElAIxlS6X1koxM2vOVAHv-MUG_IC_trcCX6K1YXyvDDPZbO4RnNOXBXRUgUaK6Apf6c9AM5-NHq67tounEbnyEqoQbMxKoxnRndm3FFaMh4CNkTbrbR7qNg_W1p5JWwvD9DuNViLj2BUt1NfahoZg">After sorting the results by descending model confidence, a brief scan of the corresponding objects makes it clear that these annotations are wrong.This may be surprising given that the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a> is being used by thousands of researchers and models, but almost all datasets have these kinds of issues.<img src="https://lh4.googleusercontent.com/iPxyYxuBrw-c2817ZQiLrCxCU_xVEyC4aHDq5044UO30_qAEeXOXc3H9PnsXxiaD9BLRI4reFCGERZ1eI4KTjVRmoCifT41hOh8mGIvOOBMwwMJhG8oQuEE2aE5iiExfkkoRRu2LLbFxiDO3OkoWYeo">Now that you have discovered some errors with the data labels, you might want to correct them.With one click, you can send the bad labels over to Scale Studio, our self-service annotation platform, where your workforce can review and fix the errors. The annotation results will be automatically imported back into Scale Nucleus, overwriting the previous, flawed set of labels.<img src="https://lh3.googleusercontent.com/JWVK3_yCVI-Z1udBltlndb0T3qWfKboWTrHFaW2IFYnIoggE2PojskkIvJfQgBecKtkJtzf5f2ti7QeYrS7Cog2mPdnhHItx2-e31Uh3ZV3HM3fqcBTBQSpSrUTcCH9cTqvZphpSoO2ulpwyOLhB-o4"><h2>What’s next:</h2>At Scale, we are perpetually focused on both data and label quality. That’s why we build tools, products, and platforms to ensure that you’re getting the best results you possibly can from your ML models. But building high-performance models starts with error-free datasets. As you are <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">exploring your dataset in Nucleus</a>, verifying label accuracy, identifying edge cases, and collecting mismatches between predictions and ground truth, you may find that you and your teammates want to collaborate on adjusting or improving your ground truth labels. If you want to make these edits in house, in the most efficient way possible, <a href="https://dashboard.scale.com/signup?product=studio" rel="noopener noreferrer" target="_blank">sign up for Scale Studio today</a>.<h2>References</h2><ol><li>Crawford, Kate. "The trouble with bias." Conference on Neural Information Processing Systems, invited speaker. 2017.</li><li>Buolamwini, Joy, and Timnit Gebru. "Gender shades: Intersectional accuracy disparities in commercial gender classification." Conference on fairness, accountability and transparency. PMLR, 2018.</li><li>Raji, Inioluwa Deborah, and Genevieve Fried. "About face: A survey of facial recognition evaluation." arXiv preprint arXiv:2102.00813 (2021).</li><li>De Vries, Terrance, et al. "Does object recognition work for everyone?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019.</li><li>Crawford, Kate, Trevor Paglen, and A. I. Excavating. "The politics of images in machine learning training sets." Excavating AI.</li><li>Northcutt, Curtis, Lu Jiang, and Isaac Chuang. "Confident learning: Estimating uncertainty in dataset labels." Journal of Artificial Intelligence Research 70 (2021): 1373-1411.</li></ol>

Nucleus helps you curate data to train better models; Studio helps you use internal resources to improve your dataset in ways that Nucleus flags.

Sometimes it's helpful to bring your own labelers domain expertise to assist in both data and model quality.

Building Great Models with Better Data on Nucleus and Studio

Learn how we extended our infrastructure by sharding our MongoDB cluster.

From MVP to Serving Enterprise Scale

How We Scaled Our MongoDB Instance Gracefully

Nucleus is the centralized, scalable, and collaborative control center for your machine learning (ML) model’s training data. Nucleus helps you update missing or erroneous labels, and identify model failure cases so that you can direct your data curation efforts with the express goal of training better models.With simple integrations for model integration, testing, validation, and deployment, Nucleus can serve as your one stop shop to continuously iterate on your dataset for superior model quality, even in the face of new edge cases, model drift, and unexpected changes in environmental variables. Nucleus helps you tackle the long tail, the one certain path to training models that can address new scenarios and eliminate failure modes.Read on to learn the essentials—how Nucleus can help you manage your dataset efficiently.<h2>The basics:</h2><h3>Addressing model failures:</h3><ul><li>Where is my model currently failing and how can I fix these issues with the next round of labeling data?</li></ul>Once you’ve uploaded ground truth labels and at least one set of predictions for your dataset, you can click into the “Objects” view in Nucleus to identify False Positives, as shown in the below video. You can also identify other simple failure modes like False Negatives or low Intersection-over-Union (IoU) scores for image data.<a href="https://scaleai.wistia.com/medias/511bisntyb" rel="noopener noreferrer" target="_blank">this video</a><h3>Addressing label quality issues:</h3><ul><li>How can I easily check and improve the quality of existing labels?</li></ul>With your ground truth labels already in Nucleus along with one set of model predictions, the IoU slider and the “sort by” drop down menu lets you sort classifications by confidence from low to high. You can then choose if you want to label or fix the lowest-confidence images, with Rapid or if you’re an enterprise customer, your labeling pipeline.<a href="https://scaleai.wistia.com/medias/bgmcf03b9u" rel="noopener noreferrer" target="_blank">this video</a><h2>Breaking down your data in new ways:</h2><h3>Finding images without labels:</h3><ul><li>How can I find images with certain properties for which no ground truth label is available?</li></ul>If no label exists for the query you’ve conceptualized, you can search your dataset with simple, natural language text. Using CLIP embeddings under the hood, Natural Language Search lets you search for images with English phrases, instead of the structured query language you use for labels and metadata.&nbsp;<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">this video</a><h3>Finding similar images:</h3><ul><li>I’ve identified an example image: how can I find all similar images in my dataset?</li></ul>If no label is present for a class of images, but you’ve found a sample image, you can use Object or Image Autotag to build up a Slice of your dataset that consists of similar images. A step-by-step process will show you how to provide positive and negative examples, in multiple rounds, to fine-tune your search for like images. When you proceed to query your Autotag, you can set a threshold to broaden or narrow the scope of your search.<a href="https://scaleai.wistia.com/medias/uuavyfeapq" rel="noopener noreferrer" target="_blank">this video</a><h2>Debug your models:</h2><h3>Find classification failures:</h3><ul><li>How can I find examples of classes in which my model and my labels disagree?</li></ul>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can find a Confusion Matrix on the Charts page. This grid lets you find color-highlighted cases in which the labels and model disagree on a classification. You can click on any box to filter your entire dataset on just those examples, to further explore where your model fails and where it succeeds.<a href="https://scaleai.wistia.com/medias/ic8y27jymv" rel="noopener noreferrer" target="_blank">this video</a><h3>Sorting and filtering both failures and successes:</h3><ul><li>How can I find the best and worst classification examples?</li></ul>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can sort all of your data objects by confidence or intersection-over-union (IOU), or bound your images with specific minimum and maximum thresholds for these score values. You can also filter by specific types of confusions such as False Positives and False Negatives.<a href="https://scaleai.wistia.com/medias/c2f14217wn" rel="noopener noreferrer" target="_blank">this video</a><h3>Avoiding regressions:</h3><ul><li>How can I ensure I'm not breaking anything that worked with a previous model or dataset?</li></ul>As you improve and expand your dataset, then proceed to train new models, you’ll want to ensure that your new models don’t fail on cases that your old models solved successfully. Through Scale Validate, Nucleus helps you track metrics such as mean average precision (mAP), precision, and recall on specific scenarios in your dataset. Or, you can establish and write your own metrics to track over time. This process ensures that every label addition and dataset modification you make only yields improving outcomes rather than regressions. You can think of this as a continuous integration (CI) or regression detection process for your models.<a href="https://scaleai.wistia.com/medias/ak5lqayith" rel="noopener noreferrer" target="_blank">this video</a> Ready to try Nucleus? <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Click Here</a> to set up your first project.

Nucleus facilitates data curation and dataset management.

Learn how Nucleus facilitates dataset curation, to help you and your team train and debug better models.

Nucleus Simplified—Dataset Management for Everyone

All companies, from the earliest startups to the largest Fortune 100 corporations, face a decision as they scale up their artificial intelligence (AI) and machine learning (ML) efforts: How much of it should be built in-house?&nbsp;That’s a particularly thorny question when it comes to <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a>. It’s important to get the approach right from the start; studies have shown that an abundance of high-quality annotated data is typically the most significant driver of model performance. But how it’s done, across all steps of the data process — from selecting a <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a> platform, to writing high-quality instructions and taxonomies, and doing the annotation itself — must be based on what works best for each unique organization.&nbsp;Here we explore what to keep in mind — the benefits and hidden pitfalls at each step of the process — as you try to solve your own data annotation build vs. buy equation.<h2>Selecting a Data Annotation Platform</h2>Choosing the right data annotation platform from the beginning of the process can minimize complications arising from changing approaches or platforms in the middle of a project.As you think about a software “home” for your data annotation activities, there are a few key capabilities you should consider:&nbsp;<ul><li>How easy is it to set up a new project or pipeline for data annotation?</li><li>How quickly and easily can annotators add high-quality, accurate annotations to your data?&nbsp;</li><li>How simple is the process to add annotators to the platform?&nbsp;</li><li>Can you analyze the annotation workforce’s performance across dimensions such as throughput, quality, and time spent?&nbsp;</li><li>What are the guardrails or in-product capabilities to ensure the data annotation output is as high-quality as possible?&nbsp;</li><li>Can you import your model output as “hypothesis annotations” to give your annotators a head-start on their work?</li></ul>Indexing on these capabilities can help guide your decision on whether to build a platform in-house or purchase a software solution. Once you’ve established your criteria in each of these dimensions, the build-vs.-buy equation comes down to a few ROI calculations:<ol><li>Customizability: If you’re looking for a platform tailor-made to suit your specific annotation needs, there will be no platform that is more customized than one that you and your team build yourself. This is doubly true if you have annotation needs that are non-standard and have minimal (if any) support from annotation platforms on the market today. The fact of the matter is, commercial tooling caters to the most common use cases, workflows, and annotation types, which may suit the majority of situations, but may not work for you. Some people customize an open source solution (such as <a href="https://github.com/openvinotoolkit/cvat" rel="noopener noreferrer" target="_blank">CVAT</a>) in order to suit their needs, and others build their own from scratch.&nbsp;</li><li>Predictability: Are your AI/ML efforts predictable over the next few years, or is it likely that you’ll experiment with different data types or annotation volumes? If you expect the scope of any AI/ML project to evolve rapidly, buying a commercial platform can be a prescient choice, since you’ll be able to spin up new data types or annotation types more quickly, provided that they’re supported. If you have a sense of data types you’ll likely experiment with in the future, it’s smart to check if they are supported in whatever annotation platform you consider</li><li>Time to market: If speed to production is an important consideration, buying an annotation platform will almost always get you there faster than building one in-house. The one exception is if your data and annotation type are simple, and supported by an open-source solution out-of-the-box. Otherwise, building a custom in-house platform can take months to get the first project in production. That’s an opportunity cost that is often neglected, but which may pose one of the greatest implicit expenses (or loss of revenue) for your organization.</li><li>Resources for maintenance: Building an in-house solution will require engineers to maintain and update it to accommodate new experiments or pipelines, and keep up with new standards and approaches. That said, even if you choose a commercial platform, you’ll still likely need to budget some engineering time to set up your pipelines – albeit less than if you’re starting from scratch – especially if you are utilizing API capabilities to import data, create projects, and/or export data.&nbsp;</li></ol>In your overall consideration, keep an eye on both short-term and long-term costs. For some smaller organizations, building an in-house tool or customizing an open-source solution “feels” cheaper because there’s no explicit cash outlay. However, you should consider whether the time and expense involved in establishing and maintaining your own data annotation tool is worth it. Future budget demands are more likely to go toward developing and training models than toward maintaining your in-house data annotation platform.For the largest organizations with custom annotation needs, the calculus becomes more nuanced. In these cases, where the team has a need for steady, large-scale annotation, where having a team of engineers dedicated to building and maintaining an in-house platform could start to make financial sense. That said, if you anticipate annotation needs evolving and quick time-to-market is important, be sure to still consider a commercial solution – at least initially, with an eye towards a potential eventual transition to in-house.<h2>Developing High-Quality Instructions &amp; Taxonomies</h2>In addition to selecting the right platform, you will also need high-quality instructions and taxonomies if you want to train high-quality models. Whether you use your in-house annotation team or outsource to a third party, explaining exactly what you want annotated (along with examples of what to do vs. not do) is critical to getting high-quality data back.&nbsp;This is an area where it makes sense to keep in-house as much as possible. You and your team will have the best, most intimate knowledge of what exactly you need annotated, as the team developing the models. By keeping this in-house, you’ll have a tight feedback loop as you scale up. You’d be surprised at how certain instructions are interpreted — and by having your ML team stay tightly involved in workshopping the instructions, providing examples, and adjusting the taxonomies, you can be confident that you're getting the best possible data back.<h2>Choosing a Workforce for Data Annotation</h2>Most teams start annotating data with internal employees who already are familiar with the data and who can start the process part-time. But as the data volume ramps up, be prepared to either recruit new full-time members to the annotation team or to outsource the work entirely to a third party.Things to consider when making this decision:<ul><li>Cost: Paying the annotation team is not the only consideration here. For a more realistic assessment of the cost of outsourcing data annotation, factor in the time and effort involved in managing an in-house team.</li><li>Flexibility: Will your data annotation needs ramp up and down quickly? Some third-party providers are prepared to handle such surges with minimal disruptions. An in-house team will need to be kept prepared to jump back in during a scaleback, while still appropriately occupied.</li><li>Subject matter expertise: While some third-party providers offer teams with specific expertise, most do not. If your annotation work is highly specialized, it may be best to recruit and train your own data annotation experts in-house.</li></ul>Some organizations choose to utilize a blend of in-house as well as external data annotators. One common workflow is using the external team to do the ‘first attempt’ of annotation, while the internal team reviews the attempts and provides an additional layer of quality assurance. Others utilize the external team to annotate steady production pipelines, and the internal team to work on data for short, bursty experiments.<h2>Conclusion: Buy Some, Customize Others</h2>Ultimately, answering the build vs. buy question doesn’t have to be an all-or-nothing proposition. If your data annotation needs are particularly custom, your budget for in-house development relatively unrestricted, and any future needs relatively predictable, you may be able to develop a great annotation solution in-house. For other teams, commercial software is likely a good choice: it can be deployed rapidly, and you can choose how and when to use its various parts. Depending on your needs and scope, third-party providers can work with your in-house teams to build or modify features and extensions as your AI/ML projects evolve.&nbsp;If you’ve considered your options and decided that you’d like to try an annotation platform built by annotation experts, <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> offers best-in-class annotation infrastructure to accelerate your in-house annotation team. Studio also offers a free tier to familiarize yourself with the platform and start annotating data. If you’ve decided you’d like to offload more parts of the data annotation pipeline, <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> provides an easy way to send your data to be annotated by Scale and get production-quality labels with no minimums.

Choosing the right data annotation approach can minimize complications down the line. What are the factors you should consider in making this build vs buy decision?

Should you build or buy a data annotation solution?

Build vs. Buy: What Makes Sense for Data Annotation?

<h3>It all started with a breakdancing video.</h3>Now, on the face of it, that shouldn’t have been too surprising, back in 2010. YouTube was already well on its way to assembling a massive collection of the human experience. We had run programs like Life in a Day - asking viewers around the world to submit videos of their daily life on a single day - to document everyday life in a way no-one had parsed before. Breakdancing and hip-hop culture were undeniably a part of that global experience - maybe surprisingly, in countries like France, where communities like <a href="https://fr.wikipedia.org/wiki/Juste_debout" rel="noopener noreferrer" target="_blank">Juste Debout </a>had started to document a thriving scene.&nbsp;But at YouTube, we had a big problem making sense of those videos. Viewers from around the world had uploaded a huge range of content, with little in the way of titles or tags to make sense of it all. For many uploaders, it was their first time uploading a video to the internet - ever. In spite of that, what made YouTube successful, and what my team focused on, was a vision to drive viewers to videos they didn’t expect, and that hadn’t amassed thousands of views - through related videos, through search, and through trends.So you can imagine my surprise in 2010 when my team showed me a 15 second video of a man spinning on his back with his sneakers in the air, on a street somewhere in Europe. The video had a random title and the uploader had no followers. Under normal circumstances, this would be a very difficult video for someone to find - just a random selection out of hundreds of thousands of videos that morning. But that day, we tried a new approach to understanding videos, based on what were called Convolutional Neural Networks, with the capacity and training horsepower to work across the whole range of the YouTube catalog.Sure enough, this time, bubbles appeared next to the video, saying “breakdancing”, “spinning”, even “<a href="https://www.google.com/search?q=breakdancing+munchmill&amp;rlz=1C5GCEM_enUS996US996&amp;ei=hCuWYtHoM7m2uvQPy9eCcA&amp;ved=0ahUKEwjRuq_M_on4AhU5m44IHcurAA4Q4dUDCA4&amp;uact=5&amp;oq=breakdancing+munchmill&amp;gs_lcp=Cgdnd3Mtd2l6EAM6BwgAEEcQsANKBAhBGABKBAhGGABQ9QJYpA1gkRFoAXABeACAAWyIAfYGkgEDNC41mAEAoAEByAEIwAEB&amp;sclient=gws-wiz" rel="noopener noreferrer" target="_blank">munchmill</a>” . Trained on tens of millions of videos, deep learning had discovered the connections between this human movement across a few frames of video, and an entire global culture built around hip-hop.&nbsp;For me, those connections launched the next decade of my career. The combination of powerful, expressive machine learning models, with new approaches to understanding the diversity and complexity of content shared on the web, was an incredible opportunity to build new experiences. My time launching a startup with Steve Chen highlighted for me how those techniques could be brought to bear to live and immersive content, while my time at Apple helped me to understand how “data centric” AI could also be used to build new hardware and software experiences with a “privacy-first” mindset.I recently joined Scale to run our engineering organization, in part because of a similar credo that Scale has in our culture - that Ambition Shapes Reality. The next decade of AI achievement is going to be built by teams that take on outsized ambitions, and understand that it is not always a straight line to get there.&nbsp;<h3 style="text-align: center;">Operational&nbsp; &amp; Data Centric AI</h3>&nbsp;The market for AI developer tools and infrastructure is currently small, and limited by the number of companies who are actually AI capable and have teams capable of using ML to drive business goals. But in the long run this market will be an industry-defining set of capabilities, and we have a golden opportunity to own the primitives that build this market, given our leadership position in autonomous vehicles and perception data. Those capabilities will include the ability to drive operational improvements through a counterfactual view of complex systems, being able to leverage multi-modal and Foundation models to work across visual and language data, and building insights in real-time as data distributions shift.Building out these capabilities for ML teams not only gives us a call option on this really exciting future, but it also gets us early and close access to the small pool of AI practitioners who are pioneering that future today. That partnership between forward-looking researchers and product leaders and engineers characterizes the fastest-moving ML teams today.<h3 style="text-align: center;">Running Through Walls</h3>When I was thinking about joining Scale, I realized I had a lot of vision alignment with Alex, Scale’s CEO. In an internal memo, Alex recently wrote, “I am tremendously excited about what’s beginning to really work here, probably more than I’ve ever been in the history of Scale.” It was obvious to me that Scale is unique, in part due to the massive opportunities in the rapidly growing emergent field of AI, and also in part by the mission upheld by leadership.&nbsp;Alex truly understands the nuances of AI. He recently did a <a href="https://www.youtube.com/watch?v=diBoo0pQ8c8" rel="noopener noreferrer" target="_blank">TED Talk at Berkeley </a>where he reiterated that AI needs humans to teach it individual values, nudge it to find thoughtful outcomes, and ensure that human intentions and values are aligned with the AI. He talked about how machines make mistakes and how they have to be taught to tell the truth.&nbsp;This alignment was a grounding factor for me. Alex understands that “AI is a supercharger for humans - when AI is better than humans, it makes humans better. AI will enable us to be even more creative and more idea driven. It allows us to embrace the generative aspect of human nature so we can run faster with our ideas and build better, more powerful solutions to the world’s biggest problems”. And building solutions to the world’s biggest problems is a focal point of Scale’s culture and formed the company’s Credos.Our credos provide a framework for us to make decisions and work effectively as a team. The one that has resonated with me the most is “run through walls.’ We are undaunted by barriers and race towards overcoming them. We step outside of our comfort zone, test our ideas, and do the hard work to get to the right solution to a hard problem. I have seen this credo in action with every one of my team members.&nbsp;<h3 style="text-align: center;">Become a Scalien!</h3>Scale is in an exciting place where growth opportunities for AI organically grow your career too. Being at Scale allows you to work with high visibility customers - often impacting millions of users.You’d have the opportunity to ideate with some of the world’s leading ML and Data Science teams at our partner companies, and find creative ways to adapt new ML techniques like Generative Models, Diffusion Models,&nbsp; and Large Language Models, or Online Optimization techniques applied to expressive embedding spaces, to building a new product.It’s been a long and rewarding journey since that first breakdancing video. I know AI and its innovation will continue to surprise me and I am excited to experience that here at Scale. I hope that you will join me here too.&nbsp;

How a breakdancing video launched a career in AI

Why I joined Scale

Data is the lifeblood of today’s enterprise, but as many organizations are discovering, it’s increasingly the quality of that data that matters more than the quantity. Highly trustworthy training data is crucial for the success of machine learning initiatives. Organizations that attempt to train models based on lower quality data are finding that accuracy ultimately suffers. In fact, with even a small amount of bad, inaccurate, or outdated data, these models are never able to become fully optimized and useful.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F830b8674f2f84bea90e7d9e6d818b766">To use an analogy, consider what happens when adults try to learn a new language. We’ve been “trained” on our native tongues for decades, and as such we’ve probably picked up a lot of preconceived notions about how language should work. Because we are ingrained with this biased training data, it’s notoriously difficult for adults to learn a new language, as we subconsciously attempt to force our old models on the process. Conversely, children who are raised bilingually have fewer of these issues; they are being “trained” with a bias-free, more balanced, dataset.We’ve seen the consequences of inaccurate and biased training data time and time again in the industry. Consider <a href="https://www.wsj.com/articles/zillow-offers-real-estate-algorithm-homes-ibuyer-11637159261" rel="noopener noreferrer" target="_blank">Zillow’s misguided home-buying algorithms</a>, which underestimated the accuracy of its home value estimates, causing it to overpay for properties or the racial bias that crept into the <a href="https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm" rel="noopener noreferrer" target="_blank">COMPAS recidivism algorithm</a>.<h2>The Importance of Data Annotation</h2>Many algorithmic problems that occur are due to poor data quality. One way in which data quality can be improved for ML algorithms is through <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">data annotation</a>, the process of tagging data with certain properties or characteristics. For example, an archive of photos of fruits could be manually annotated as apple, pear, watermelon, and so on, giving an algorithm a head-start when it comes to predicting the identity of other, unlabeled objects. <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">Data annotation</a> can be a tedious, manual chore, but as datasets become very large and complex, annotation can become crucial.The consequences of poorly annotated data can be not only annoying but also costly, as models must be redesigned, retrained, and run over and over again. In some cases, the consequences can be devastating if an organization makes decisions based on a badly trained ML model, such as a medical algorithm designed to spot signs of cancer in an x-ray.<h2>How Good Is Your Training Data?</h2>So, let’s say your data is already annotated. Is it any good? How can you find out?Start by assessing the performance of your model. It’s often straightforward to discern whether results are in line with expectations. If your fruit-detection algorithm is coming up with nothing but bananas, you might have a data problem, and some deeper investigation is probably in order.A good next step (either way) is to take a random sample from your dataset and meticulously audit its annotations. How accurate are the annotations? Are annotations missing? This kind of effort can give you a good sense of how accurate your data will be for a specific task as well as a look at the overall health of the project – assuming the sample you took is truly random. If you are looking for more automated ways of debugging your dataset, tools like <a href="https://scale.com/nucleus" rel="noopener noreferrer" target="_blank">Scale Nucleus</a> can help identify errors and gaps in your existing data.<h2>How to Improve Data Annotation Quality</h2>If you’ve uncovered a problem with the quality of your data annotation, you can remediate the problem via a number of tactics, including: <ol><li>Make sure your annotation instructions are well-understood. Are annotators incorrectly including the stem of the fruit or the container in which the apples are in as part of the annotation? If they aren’t following instructions perfectly, your annotation efforts will be wasted – and your data may be worse off when the task is finished than when you began. Start slowly by giving annotators a small number of easier tasks, allow them to ask questions, and ensure they understand what makes for a high quality annotation. As you work through the training process, revise your instructions to be as clear and precise as possible, which will help you with the next annotator.</li><li>Add a review cycle. Add a second layer of personnel responsible for overseeing the first-round of annotators. (This second layer of reviewers can be chosen from those annotators who prove themselves adept at high-quality annotations.) This second level does not create new annotations from scratch but rather monitors annotation work, corrects any errors as they are spotted, and adds annotations that might have been forgotten somewhere in the process. This extra check layer can increase the overall quality of your data considerably.</li><li>Try a consensus pipeline. One solid quality tactic is to have multiple people annotate the same data, then use consensus to determine which annotation is correct. If four people tag a fruit as “apple” and one tags it as “orange,” it’s highly likely that the three apple tags are correct and the orange tag can be discarded. If this pipeline also considers the overall accuracy level of each individual annotator, you can build an even higher confidence in your data quality.&nbsp;</li><li>Add a quality screen for annotators. If annotators want to work for you, put them through a quality test first. They have to achieve a set accuracy – 99 percent, perhaps – or else they aren’t allowed to start working on your queue. This step saves you from having to rigorously monitor quality and ensures that you’re starting with high-quality annotations.</li><li>Add evaluation tasks throughout the queue and evaluate your annotated data against this benchmark. Evaluation tasks are tasks that have known correct answers. This benchmark serves as a kind of answer key against which an algorithm can be tested, allowing you to determine the quality of your data without having to manually check every piece of the dataset. When data is being annotated, each annotator should be directed to complete the same set of benchmarks, giving you an ongoing understanding of your dataset’s average quality level. By periodically benchmarking your data, you can ensure nothing has gone off the rails and that annotators are still on target, even after the initial training period has passed. If annotators’ evaluation scores are dropping, you may want to provide more in-depth oversight and training.</li></ol>To get your data annotated with high quality, you can either leverage your own team of annotators - or you can have Scale’s team of trained annotators do it for you.&nbsp;<ul><li><a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> is a data annotation platform for you to use with your own workforce, and it comes with all the tools to implement the best practices listed above, learned from providing billions of high-quality annotations for our customers. If you are training and managing your own team of annotators, Studio provides the annotation tools and the management systems to achieve a high-quality data set.&nbsp;</li><li><a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> allows you to send your raw data to Scale, and we’ll return high-quality data quickly using the best practices listed above. Simply define the parameters of your project, and send your data out for annotation. There’s no need to reinvent the wheel when you can simply pay as you go and receive production-level data in hours instead of weeks.</li></ul>

Training data is the lifeblood of machine learning. Learn how to assess the quality of your annotated data and methods to develop high-quality training data.

Understand and improve the quality of your training data. 

How to Understand and Improve the Quality of Annotated Data

When setting up a new machine learning (ML) system, collecting and annotating data can be a time-consuming process that requires a significant upfront investment. Since data quality is strongly correlated with ML model performance, it’s crucial to invest in high-quality solutions for data annotation from the onset of your project. The process of annotating data for an ML project can be surprisingly complicated – There are a variety of factors you’ll need to consider, from how you’ll manage your approach to annotation to what tooling and automation approaches you’ll use.<h2>How to Build Your Data Annotation Pipeline</h2>Annotating data at high quality requires significant upfront and continued investment over the lifespan of your ML system. While there are automated solutions to speed up and streamline the data annotation process, having humans-in-the-loop is key to producing high-quality results. When setting up a data annotation pipeline, your company will need to decide how you’ll find and train annotators, as well as how you’ll measure their performance. To ensure annotators produce high-quality annotations, you’ll need to provide them with performance incentives that align with your most important annotation metrics, whether that’s throughput or quality, or a combination of both. You’ll also need to provide annotation tools for your workers, as well as design a pipeline that encourages your annotators to output consistent, high-quality annotations.<h3>Hiring Annotators While Minimizing Overhead</h3>Finding annotators to complete your task is one of the first challenges you’ll face in the data annotation process. You’ll need to identify workers who are well-suited to annotate your data, whether that involves selecting people who speak a certain language, are from a specific location, or have a particular type of expertise. You’ll also need to decide how to approach the hiring process. Your choice of workforce influences whether you have access to a large selection of annotators who work fewer hours, or a small selection of annotators who work longer hours and generate more consistent results. If you expect a continuous flow of production-quality data, you’ll also need to potentially hire workers across time zones to meet the demands of your project.&nbsp;You should decide whether to hire a larger pool of diverse annotators versus a smaller set of consistent annotators based on your use case. If you anticipate that you’ll need annotations for small, irregular projects, for example, you’ll likely want to rely on a larger selection of annotators that can accommodate short-term surges in need. Alternatively, if you’re pursuing a long-term production project with consistent annotation needs, a smaller team of annotators with regular annotation hours may deliver better results.Once you’ve hired annotators, you’ll also need to devote resources to communication and expectation setting to ensure that your annotators provide you high-quality results. In short, just finding workers to annotate your data and keeping them on your platform can be a time-consuming process with logistical overhead! These challenges are often why many ML teams opt to outsource the entire process to an external company.<h3>Training Annotators and Monitoring Performance</h3>Once you’ve hired your annotators, your work is not yet over – You need to provide training to teach them to accurately annotate your data across different use cases, task types, and annotation types. To ensure high-quality annotations, you’ll need to provide workers with clear and comprehensive instructions, so that different workers can annotate data consistently. You may also find over time that there are several edge cases within your data that your instructions do not cover, as it can be difficult to write non-ambiguous and comprehensive instructions that cover all potential edge cases.As the administrator of the data pipeline, you’ll need to create a “golden dataset” that has accurate annotations for a subset of your data. You can use this dataset to check that workers understand the instructions, as well as to periodically re-evaluate their performance against a gold standard. You should also regularly update this dataset with new examples to provide annotators with up-to-date examples that reflect real-world data.&nbsp;Suppose your goal is to achieve 95%+ quality in your overall dataset. If all your annotators consistently achieve 95% quality, you should be able to achieve this goal. However, if there is a particular edge case that only appears 1% of the time, your annotators can get 100% of that 1% wrong and still average 95% quality overall. While you are still hitting your goal of 95%+ quality, this is not ideal, as your model will not be able to account for this edge case if trained on this dataset.To mitigate this challenge, you will need to add examples to your golden dataset to ensure that it remains well-balanced across all classes and edge cases. By updating and adding new examples to your dataset, you can effectively assess annotator performance over time, and achieve a holistic view of their performance across various classes and edge cases.<h3>Reviewing Annotations for Quality</h3>After your annotators complete tasks, you’ll need to evaluate the final output. To do this, there are two main methods for evaluating annotation quality. For classification or text-based tasks where there is more subjectivity, you can implement a consensus algorithm. Suppose you want to develop a sentiment analysis model, you can have multiple people determine if a person in an image looks happy or sad and the option the majority selected will win. If there isn’t enough confidence in the voted result, you can have a higher-level reviewer determine the final result. In some cases, you may also not want to weigh every person’s vote the same. If you have higher confidence in some people’s votes over others, you may want to implement a weighted voting system that biases the final response more towards the votes of people you have higher confidence in.&nbsp;For image-based or other computer vision tasks, you can have one preliminary attempter annotate the task, then have a higher-level reviewer either review the work and accept the results or review the work and fix annotations.&nbsp;You can select reviewers from past annotators who consistently produce high-quality results. Selecting reviewers, however, can be complicated, since someone’s annotation performance can fluctuate over time. You’ll need to constantly identify reviewers who are no longer performing well so that you can demote them to attempts, while also identifying well-performing annotators so that you can promote them to reviewers. Since this approach removes high-quality annotators from the labeling pool, it will naturally reduce your throughput, so you’ll want to find a balance between quality and throughput.Whatever approach you follow for evaluating label quality, you’ll need to be on the lookout for people who consistently underperform and label data poorly. You should automatically identify underperforming annotators, and either retrain them so they produce higher-quality labels, or remove them from the project entirely.<h3>Selecting the Best Annotation Tool For Your Task</h3>Another big decision you’ll need to make is on what labeling tools your workers will use. You can either build your own tool set or use available commercial tools. While you can specialize in-house tools for your specific task, they can be difficult and time-consuming to build, especially if they need to support different annotation types. For example, image, audio, and text data all require different annotation approaches – For image annotation, you’ll need a system to draw boxes, polygons, or other geometries around identified objects; whereas, for audio annotation, your system will need to support transcription and associating timestamps with event attributes. For classification tasks, on the other hand, your system will need to permit multiple-choice selections and potentially large taxonomies. For video annotation tasks, video files can be very large in size, making them technically challenging to support, especially if your workers have less powerful laptops or slow internet connections.&nbsp;&nbsp;Compared to in-house tooling, commercial data tools make it easier to jump into labeling your data, although they may be less specialized for your use case. Another consideration when selecting the right annotation tool is ML-assisted tooling. Some tools may have ML-assisted tooling, like our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosemseg tool</a>, to improve annotation efficiency and speed while others do not. Building and selecting the appropriate labeling tools is vital for optimizing labeler efficiency, so you should ensure the tools you select are well-suited for your use case.<h2>What Next?</h2>Needless to say, getting data annotation right can be a challenging and time-consuming process. You’ll need to consider a wide range of factors, from how you’ll manage your labeling workforce to what automation approaches you’ll use. Your data labeling pipeline is a process that’s incredibly important to get right since ML system performance is correlated with data quality.Instead of building a data pipeline from scratch, you can leverage a product like <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> or <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> to quickly create production-level ratings. It’s designed so that you can set up your projects in minutes and get high-quality labels within hours. Trusted by AI teams like OpenAI, Square, and Pinterest, this tool can help you hit the ground running on your next ML project!

Data annotation, or data labeling, is a more challenging problem than most teams realize but investing in annotation is key to unlocking the power of AI.

Scale AI’s How To Guide to Data Annotation/Data Labeling

How to Setup Data Annotation Pipelines

At first glance, human-in-the-loop processes seem like a relic of older times. Before technological advancements, humans effectively performed functions completed by computers today. For example, NASA hired large teams of arithmetic processors- men and women whose only job was to evaluate numerical expressions quickly and accurately. These calculators were pivotal to enabling space exploration.At Scale, our labeling teams are just as critical to achieving our mission. We’ve found we’re able to effectively apply the principles behind generative adversarial networks to our human-in-the-loop processes to deliver the highest quality annotations to customers.<h2>How does this work, exactly?</h2>A GAN, or generative adversarial network, is given an unlabeled dataset and learns to produce new examples that are not already in the dataset but are indistinguishable from the items in the dataset. For example, a net meant to produce new images of flowers would be fed as input images of real flowers. To achieve this goal, the net has to achieve two things:<ol><li>The generated flowers must not be distinguishable from real flowers. That is to say, if given a set of both generated and real flower images, one should not be able to tell which is which.</li><li>In achieving the above, the GAN must not simply produce copies of the real flower images.</li></ol>Therefore, the net must be trained to generate new flowers, discriminate between real and fake flowers, and avoid simply replicating training data. The training loop jointly optimizes the generator (which produces new images that are not copies of the training set), and the discriminator (which attempts to discriminate between training data and generated images). The net is successful when the generator becomes accurate enough to fool the discriminator, i.e. the discriminator cannot outperform random choice when classifying generated vs non-generated examples.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F018ccca0549b448bb3686f7d5703244e">In the context of human-in-the loop processes, we can apply this same principle to produce quality labels on subjective tasks. Let’s say the pipeline we’ve chosen has a minimum of two annotators seeing each task, one attempter and one reviewer.Example Task:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F4a26d7e4818f49dfa8acd7616a2bf50c">Instructions: Please describe the image as accurately and descriptively as possible with a short phrase.Possible attempter responses: <ul><li>Attempter 1: ‘asdf’</li><li>Attempter 2: ‘trees’</li><li>Attempter 3: ‘green hills’</li><li>Attempter 4: ‘grassy hills with mountains, trees, houses’</li><li>Attempter 5: ‘grassy hills with trees and 3 houses, mountains and clouds in background’</li><li>Attempter 6: ‘tall trees on green grassy hill, small red and gray houses, blue mountains in background’</li></ul>The attempters act as generators, and can generate any one of the above responses for this task. Attempters 1-3 have written low-quality responses. On a regular rapid project, these attempters would be disabled via evaluation tasks. However, on a generative task like the above, exact-match evaluation tasks would not guarantee quality. For this image, both Attempters 5 and 6 provided accurate and descriptive responses. Therefore, we turn instead to the discriminator to filter out the better responses.Our discriminators are reviewers. A reviewer is the final person to look at a task and is able to see the attempters’ responses. Reviewers are the final stage before tasks are returned to the customer If we allow reviewers to edit an attempter’s response directly and write a caption themselves, we run into the same problem as with attempter evaluation tasks.&nbsp;However, once we move more into thinking of the reviewer as a GAN discriminator, the solution becomes clear. Reviewers must discriminate between good and bad responses, but they need not write any captions themselves. Although there is no 1:1 mapping between a task and a certain correct answer, we do know upon encountering a caption whether it is acceptable or unacceptable. So we give the reviewers the choice of either accepting or rejecting an attempter’s response, but we do not give them the option to change it. Now the process of evaluating their quality is simple- create evaluation tasks with pre-generated responses that are expected to be accepted or rejected appropriately.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb8be702f8ef547cba93bf016ee11d500">The two parts of a GAN, generator and discriminator, work together to pass information between them. In our human network, reviewers may provide feedback upon rejecting an attempt, to help the generator learn. The discriminator, composed of both the reviewers and the reviewer benchmarks managed by the customer, learns in turn from the generator. From a variety of generated data, the customer and in turn the reviewers are able to converge on a similar understanding of the desired result.Now that we are sure the discriminator is accurately discriminating between low and high-quality responses, we are able to train the generator. What this means is that attempters are now disabled based on how many of their responses are rejected. Over time, this leads to a state where the attempters are effective, high-quality generators- essentially a trained GAN.This pipeline is available via Rapid’s <a href="https://docs.scale.com/reference/text-collection" rel="noopener noreferrer" target="_blank">text collection</a> and <a href="https://docs.scale.com/reference/categorization" rel="noopener noreferrer" target="_blank">categorization</a> projects. <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Get started with Scale Rapid</a> today!

Applying the principles behind generative adversarial networks (GANs) to our human-in-the-loop data annotation processes.

Using generative adversarial networks (GANs) to align human metrics

Using GANs to Align Human Metrics

At<a href="https://scale.com/defense" rel="noopener noreferrer" target="_blank"> Scale</a>, we are committed to partnering with the U.S. government and technology leaders to ensure U.S. investments in AI solve critical challenges. It’s imperative the U.S. government has access to, and is deploying, the best quality data and AI technology available.&nbsp;That's why we are excited to announce Scale has partnered with <a href="https://www2.deloitte.com/us/en.html" rel="noopener noreferrer" target="_blank">Deloitte</a>, an awardee of the Joint Artificial Intelligence Center's Data Readiness Artificial Intelligence Development (DRAID) Basic Ordering Agreement (BOA). The JAIC DRAID is a five-year, $241 million BOA that enables Scale and Deloitte to support AI data preparation for the U.S. government. This partnership brings nearly $500M in&nbsp;<a href="https://www.businesswire.com/news/home/20220131005304/en/Scale-AI-Selected-by-U.S.-Department-of-Defense-to-Accelerate-Government%E2%80%99s-AI-Capabilities" rel="noopener noreferrer" target="_blank">total JAIC ceiling value</a>&nbsp;to Scale. Scale’s technology will be readily available to all federal agencies and will empower the government to feel confident in leveraging AI and ML in their most important workflows.With this partnership, Scale will provide services focused on data management, AI modeling and innovative development including:<ul><li>High quality data integration, prep, labeling and management</li><li>Synthetic data generation&nbsp;</li><li>NLP-enabled processing</li><li>Algorithm management</li><li>Innovative AI application development</li></ul>Scale’s data-centric approach to its end-to-end AI platform provides the U.S. government with the support needed to manage the entire AI lifecycle, all while ensuring the highest quality data is incorporated. Using our technology, analysts and operators will be able to understand, visualize, curate, and collaborate on their data through process automation, enhanced decision support, and insight generation, faster and with greater accuracy.

Scale has partnered with Deloitte, an awardee of the Joint Artificial Intelligence Center's Data Readiness Artificial Intelligence Development (DRAID)

Scale AI Partners with Deloitte

Scale partners with Deloitte to accelerate AI adoption across the U.S. government

I spent the last 6 years of my career at Uber across both Finance and Business leadership roles. In this time, I witnessed first hand how AI and ML drove improved customer experiences, more efficient operations, and will be instrumental in unlocking a future with self-driving cars and trucks.&nbsp;As I considered joining Scale AI, I got deeper into the latest trends and potential of AI. I have become even more excited about the many applications of AI and AI’s ability to improve decision making, drive productivity leaps, as well as deliver improved customer experiences and outcomes across every industry and domain on the planet.&nbsp;What I found in Alex and his team at Scale is a vision to help all companies and the U.S. government realize the potential of AI. Scale’s mission is to accelerate the development of AI applications and to do it by first tackling one of the hardest problems in AI, better data. Scale started by supporting the autonomous vehicle industry but has since expanded across many industries and applications. They have built products and solutions that support customers at the leading edge of AI as well as companies that are just getting started. In short, this convinced me that the opportunity and potential are unparalleled.&nbsp;As I spent more time getting to know Alex, we talked about what he was looking for in a CFO. We agreed that Scale needs a business-minded finance partner to foster continued, rapid growth of the Finance function that enables the company through increased strategic alignment, business model evolution, and increased operational discipline.&nbsp;Today, I am excited to announce that I am joining Scale as their first CFO to help the team deliver on their vision of building and scaling what will be one of the most impactful enterprise companies in the world by accelerating the adoption of AI. I will be leading Finance, Accounting, Corporate Development, and Strategic Partnerships. We are hiring across the team. Come join us!&nbsp;

Blog

Engineering

How to Fine-Tune Llama 2 With LLM Engine

Product

Scale Unlocks Open-Source LLMs With New Platform and Partnership with Meta

Company

Scale AI chosen by the U.S. Army for Robotic Combat Vehicle Program

Engineering

Scale's Solution for Aerial Object Tracking

People

Why I Joined Scale

Product

A letter from our CEO

Product

Scale + Anthropic: Full-Stack Generative AI for Enterprises

Company

Scale’s DISA IL-4 Provisional Authorization Increases Accessibility to AI for the Department of Defense (DoD)

People

Women’s History Month at Scale

Engineering

How to Fine-Tune Llama 2 With LLM Engine

Product

Scale Unlocks Open-Source LLMs With New Platform and Partnership with Meta

Company

Scale AI chosen by the U.S. Army for Robotic Combat Vehicle Program

Engineering

Scale's Solution for Aerial Object Tracking

People

Why I Joined Scale

Product

A letter from our CEO

Product

Scale + Anthropic: Full-Stack Generative AI for Enterprises

Company

Scale’s DISA IL-4 Provisional Authorization Increases Accessibility to AI for the Department of Defense (DoD)

People

Women’s History Month at Scale