SCREEN OPERATION RESEARCH
Teaching AI to fill stock order forms, then reducing its size.
Recorded: 15 October 2026. Model evaluations: 13–14 October 2026.
We trained Clef to operate a mock stock order form and reduced its size. We then checked whether it could fill the fields as instructed.
References: Qwen · LoRA · NVFP4 · Order types · IFDOCO · SOR
The AI receives the order conditions and screen information. Clef scores the possible actions, and the app enters the values. It reads the screen again after each action to check the result. It hands control to a person before password entry.
We used browser-screen records for training. LoRA trained the language layers; the image reader and action-scoring head stayed fixed. Clef and Jev-Omni were trained and quantized separately.
Five examples from the training split of “Order Form Training Data v3,” now managed by ML Foundry. Each pairs order conditions, a screen and possible actions with its training target.
All five use fictional order conditions and synthetic mock screens. The broker name is covered in white in the published images. The dataset name is a publication alias. order-forms-v3 / train
Training target · option_00
Set quantity to 100
{
"kind": "SET",
"field": "input_quantity",
"value": "100",
"tab": "normal"
}Source record: sbi-train-normal_limit-57afea5d1dee-step-04
Training target · option_02
Set OCO trigger price to 1220
{
"kind": "SET",
"field": "oco2_input_trigger_price",
"value": "1220",
"tab": "oco"
}Source record: sbi-train-oco-9707d3b94b9d-step-08
Training target · option_03
Set IFD exit limit price to 1250
{
"kind": "SET",
"field": "done_input_price",
"value": "1250",
"tab": "ifd"
}Source record: sbi-train-ifd-7b0094b34c69-step-12
Training target · option_02
Hand control to a person
{
"kind": "HANDOFF",
"field": null,
"value": null,
"tab": "ifdoco"
}Source record: sbi-train-ifdoco-1d6aa9d327b7-step-16
Training target · option_00
Hold without entering values
{
"kind": "NO_ACTION",
"field": null,
"value": null,
"tab": "normal"
}Source record: sbi-train-normal_limit-fba79fff7f57-step-00
The current setup combines DOM screen information with an image. We evaluated 337 decisions and 30 order conditions on mock screens with the correct action included among the options. There were 24 valid conditions and six invalid or incomplete ones.
The six order types were limit, market, stop, OCO, IFD and IFDOCO. We counted individual correct decisions and complete input sequences separately.
| Clef version | Correct decisions | All fields correct and human handoff |
|---|---|---|
| Before fine-tuning (BF16) | No quantitative score | No quantitative score |
| After the first training run (BF16) | 322/337 (95.5%) | 16/24 (66.7%) |
| After the second training run (BF16) | 337/337 (100%) | 24/24 (100%) |
| After quantization (NVFP4) | 337/337 (100%) | 24/24 (100%) |
The first run was evaluated using screen information from saved records plus images; the second run and quantized version used browser DOM plus images. Both training and input format changed in this comparison. There is no quantitative score from before fine-tuning. At that stage, DSH session logs showed unsuccessful behavior, including runs stopping partway through.
In the quantized setup, the six invalid or incomplete conditions produced a decision to withhold input. All 30 records were complete. Attempts and actions beyond the stopping point both totalled zero.
| Configuration | Weight size | Share of original size |
|---|---|---|
| After the second training run (BF16) | 54.97 GB | 100% |
| After quantization (NVFP4, mixed precision) | 23.16 GB | 42.1% |
Weight size fell by 31.81 GB, a 57.9% reduction. At about 42% of the original size, the model retained 337/337 correct decisions and 24/24 complete inputs with human handoff in the mock-screen evaluation above.
Sizes are calculated from the saved tensor metadata, including the model and final decision head. One GB is one billion bytes.
Once the quantized Clef model was ready, we sent 336 requests one at a time. Median response time was 0.978 seconds. About 95% returned within 1.227 seconds.
For the second training run, we made two inputs from each example: screen information with an image, and an image-centred view. Their training weights were 50:50.
The smaller model uses NVFP4 for MLP, FP8 for attention, and BF16 for image reading and the final decision. The serving paths use different option numbers, so we compared the actions they represented.
We gave DSH written order conditions and let it fill the mock form. A separate screen reading confirmed that each field matched the instructions in these two examples.
Summary of the recorded run
Requested conditions
Buy fictional stock 0001 on the Tokyo exchange: 200 shares, limit price 1234.5 yen, valid for the day, no price condition, no SOR and no points.
A separate screen reading confirmed the values
Requested conditions
Sell fictional stock 0004: 500 shares, limit price 3580 yen; at 3320 yen or below, revise to 3310 yen. Valid for the day, with no SOR.
A separate screen reading confirmed the values
This check used 30 decisions and 29 input operations. It sent 59 image-backed requests to Clef.
Qwen Flash Next with low reasoning handled the request. Clef NVFP4 judged the screen actions. The browser and app routes used the same Brave browser.
The video shows a separate run recorded on 15 October. It starts with quantity 200 already entered and shows the limit price, day validity and points setting. The price-condition field still reads "Select".
User recording, 35.8 seconds, silent, original speed. The excerpt places the form and operation log one above the other.
Open the MP4 directlyAdding form-term explanations to Jev-Omni BF16's input increased correct decisions from 270/337 to 322/337. Completed input and human handoff rose from 2/24 to 18/24. A separate 18-case condition check changed from 10/18 to 9/18.
We compared quantization separately using the earlier model. BF16 gave 257 correct decisions out of 337; NVFP4 gave 250. We saved the converted model, loaded it again and checked that it ran.
The checks run on macOS in UTM. The app compares control roles, positions and values with the image, then operates the verified target.
The sequence alternating five browser operations and five app operations remains under evaluation.
We tried Cua Spaces after using DSH with UTM, then returned to UTM. It made screen-information handling and the connection to the decision model easier to adjust locally.
Clef receives a diagnostic request when screen information is unavailable. Input permission is checked separately, and the app proceeds once it has verified the target.
Saved results for Clef's decisions and order-input evaluation.
The image below shows backtest results comparing six trading strategies on historical stock prices. Performance, trade records and profit and loss appear together on one screen.
The same-day live pipeline learns from the incoming stock-price stream, analyzes it every three minutes, and generates trade records in two minutes.
In the planned automation, a Familia agent pulls the trade records and fills in and configures the order screen accordingly. A person then enters the password and presses the order submission button.
Operating the order screen, including input and settings, currently takes about 50 seconds per order. We are considering fine-tuning Qwen3.8 to reduce this operation time further. The workflow is intended for trading styles other than scalping, with screen operation within one minute as our practical target.
Excerpt from a backtest screen covering 20 trading days, 9 September–9 October 2026. Under evaluation.
View larger image