How to Import CSV Into Amazon S3

Amazon S3 documents several ways to move a CSV into a bucket. The console takes a file by drag and drop, and its own page caps that at 160 GB. The AWS CLI and the SDKs send "a single object up to 5 GB in size" in one PUT, and multipart carries one object to 50 TB. AWS Transfer Family answers SFTP, FTPS and FTP straight into the same bucket. Each one starts from a seat that already holds AWS credentials. Your customer sits outside that seat, holding the harvest export a research station emailed over last night.

What S3 checks

Nothing about the bytes. There is no schema on a bucket, no column type, no constraint and no unique index. A CSV with a missing field, a stray line break or a date in the wrong century arrives with a 200 OK and stays.

Two things are checked, and both are worth building on.

The key. An object key is "the unique identifier for an object within a bucket", up to 1,024 bytes of UTF-8. AWS lists the characters that are safe in one, and lists the ones it recommends against.

The precondition. Send If-None-Match with the * value and S3 refuses to write over an object that already exists, answering 412 Precondition Failed.

What S3 declines to offer matters as much. "Updates are key-based. There is no way to make atomic updates across keys." An import spread over several objects has no transaction behind it, so the batch needs a shape of its own.

The file that arrives

A seed company runs variety trials through contract research stations. Each station sends one export at the end of the season.

1Plot,Site,Cultivar,Harvest Date,Yield (t/ha),Moisture %,Fungicide,Notes2TR-2026-0148,Wageningen (NL) #2,Belmonte,14/05/2026,"9,84","14,2 %",Untreated,Lodging on the north edge3TR-2026-0149,Wageningen (NL) #2,KWS Extase,21/10/2026,"10,20","13,9 %",standard,"Sprayed late,
weather delay"4TR-2026-0150,Wageningen NL 2,Belmonte,03/09/2026,,"15,1 %",Enhanced Fungicide,Yield sheet missing5TR-2026-0151,Rothamsted (UK) #1,Champion,21/10/2026,"8,70","12,8 %",none,⋮91 rows not shown97TR-2026-0243,Rothamsted (UK) #1,KWS Extase,29/10/2026,"9,15","13,4 %",standard,

The header row says Cultivar where the company says variety, and Fungicide where it says treatment. One station appears as Wageningen (NL) #2 on three rows and Wageningen NL 2 on the fourth. Dates run day first. One yield cell is empty, and one notes cell carries a line break inside its quotes.

The person drags the file into the importer inside your app. Updog Importer reads it in the browser, matches the headers to your schema, holds the sites and treatments to your lists, and puts every row in front of them. Your onComplete handler posts the clean rows to a route you own. That route writes them into the bucket as objects.

No Updog server stands between the browser and S3.

The layout in the bucket

S3 stores no folders. "The Amazon S3 data model is a flat structure", and the tree the console draws comes from prefixes and the / delimiter inside key names. So the layout is a naming decision, and it is the closest thing this destination has to a schema.

s3://acme-trials/
  harvest/
    season=2026/
      site=NL-WGN-02/
        batch=9f2c1e84-7d3a-4c11-b0a6-2e5f7c9d1b40/
          part-0001.csv
          part-0002.csv
      site=UK-RTH-01/
        batch=9f2c1e84-7d3a-4c11-b0a6-2e5f7c9d1b40/
          part-0001.csv
  _batches/
    season=2026/
      9f2c1e84-7d3a-4c11-b0a6-2e5f7c9d1b40.json

One import becomes one batch id, and every object it writes carries that id in its key. The parts sit under the season and the site, which is what lets a query engine read one station's 2026 harvest without touching anything else. The manifest sits outside the prefix the table reads, so nothing has to explain a JSON file to a CSV table.

Splitting one import across several keys costs nothing here. S3 handles "at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix", and there are no limits on the number of prefixes in a bucket. Rows destined for a warehouse take a different road, and importing a CSV into Amazon Redshift shows that one.

The value that becomes part of the key

Wageningen (NL) #2 is a site name in a spreadsheet. It is also, one step later, a piece of a path. It carries a space, which AWS lists among the characters that "might require additional code handling", and a pound sign, which sits in the list AWS recommends against "because of significant special character handling, which isn't consistent across all applications".

S3 would take it. Every tool that reads the bucket afterwards is the thing at risk.

So the site column is a closed list of station codes, and the messy spellings reach those codes through value matching. NL-WGN-02 holds only characters from the safe set, and both spellings in the file arrive at it. The route checks the code against a pattern a second time before it builds a key, since a value from a browser is a value from a browser.

The schema in Updog Importer

The columns array carries the shape the bucket will hold.

import type { DataEditorColumn } from "@updog/data-editor";

const SITES = ["NL-WGN-02", "UK-RTH-01", "FR-CLF-03"];
const TREATMENTS = ["Untreated", "Standard fungicide", "Enhanced fungicide"];

export const columns: DataEditorColumn[] = [
  {
    id: "plotRef",
    title: "Plot reference",
    size: 160,
    transformer: (value) => String(value).trim().toUpperCase(),
    validators: [
      { type: "required" },
      { type: "regex", pattern: "^TR-\\d{4}-\\d{4}$" },
      { type: "unique" },
    ],
  },
  {
    id: "site",
    title: "Trial site",
    size: 150,
    editor: { type: "select", options: SITES, enableCustomValue: false },
    validators: [{ type: "required" }, { type: "oneOf", values: SITES }],
  },
  {
    id: "variety",
    title: "Variety",
    size: 160,
    validators: [{ type: "required" }],
  },
  {
    id: "harvestedOn",
    title: "Harvest date",
    size: 150,
    editor: { type: "date" },
    validators: [{ type: "required" }, { type: "date" }],
  },
  {
    id: "yieldTHa",
    title: "Yield",
    size: 120,
    editor: { type: "number" },
    validators: [
      { type: "required" },
      { type: "number", min: 0, max: 30, decimalPlaces: 2 },
    ],
  },
  {
    id: "moisturePct",
    title: "Moisture at harvest",
    size: 180,
    editor: { type: "number" },
    validators: [
      { type: "required" },
      { type: "number", min: 0, max: 100, decimalPlaces: 1 },
    ],
  },
  {
    id: "treatment",
    title: "Treatment",
    size: 190,
    editor: { type: "select", options: TREATMENTS, enableCustomValue: false },
    validators: [{ type: "required" }, { type: "oneOf", values: TREATMENTS }],
  },
  {
    id: "notes",
    title: "Notes",
    size: 260,
    transformer: (value) =>
      String(value ?? "").replace(/[\r\n]+/g, " ").trim(),
  },
];

Two rules exist because of what reads the object later, and the section below names the pages they come from. required on both numeric columns keeps an empty cell out of a numeric column in the table. The transformer on notes collapses any line break the station left inside a quoted field.

The rest guard the key and the lists. The regex holds the plot reference to one shape, unique catches the same plot twice in one file, and the two select editors with enableCustomValue: false refuse anything outside the station codes and the treatment names.

The headers the station sends

Updog scores each header against the column id and the column title, and the higher score wins.

Harvest Date is the one to watch. Against the column id harvestedOn it scores nothing, since the two share no whole word and sit too far apart for the edit-distance tier. Against the column title "Harvest date" it normalizes to the same string and lands at a hundred.

Cultivar and Fungicide reach nothing on their own. Neither shares a word with the column it belongs to, and no built-in group carries either. Both are one line each in the synonyms block below.

Values follow the same ladder. Enhanced Fungicide normalizes onto the option. standard is contained by Standard fungicide. none and the two site spellings are spelled out in the mount.

The file settles its own dates. 14/05/2026 on the first row carries a value above twelve in the first position, so the scan stops there and reads the whole file day first, and 21/10/2026 and 03/09/2026 follow. The numbers work the other way. Each of 9,84 and 14,2 % carries one separator, so each one is a vote, and the majority settles the file as European.

The mount

The props tie the file, the schema and the bucket together.

<DataEditor<TrialRow>
  apiKey="your-license-key"
  open={open}
  onClose={closeEditor}
  columns={columns}
  primaryKey="plotRef"
  blockSubmitOnError
  remoteSources={lastSeason}
  synonyms={{
    columns: {
      variety: ["cultivar", "line"],
      treatment: ["fungicide", "spray programme"],
    },
    values: {
      "NL-WGN-02": ["wageningen (nl) #2", "wageningen nl 2", "wgn2"],
      "UK-RTH-01": ["rothamsted (uk) #1", "rothamsted 1"],
      Untreated: ["none", "nil", "control"],
    },
  }}
  onComplete={onComplete}
/>

primaryKey is the plot reference. Its work here is identity inside the import, because the destination answers no question about what it already holds. S3 lists keys and returns objects. Asking whether plot TR-2026-0148 is in the lake means reading objects back through a query engine, so this import loads nothing and every row arrives new.

blockSubmitOnError keeps submit disabled while any row carries an error, which is what keeps the empty yield on row three out of the bucket. Whatever the person fixes by hand comes back as learnedSynonyms, ready to store and feed through synonyms next season.

Every snippet here is React. Those props reach Vue, Angular and Svelte through the web component build. The install and the modal wiring beneath this mount live in how to import a CSV file into a React app.

Last season's object, back in the grid

The bucket is where last season's export already sits, so the upload step can offer it as a starting point. remoteSources renders a button and hands you the fetch.

const lastSeason = useMemo(() => [
  {
    id: "last-season",
    label: "Last season",
    icon: "<svg>...</svg>",
    description: "Open the 2025 harvest file for this site",
    fetch: async () => {
      const answer = await fetch("/api/trials/last-season");
      if (!answer.ok) throw new Error("That season is not in the bucket yet");
      const body = await answer.blob();
      return new File([body], "wgn-2025-harvest.csv", { type: "text/csv" });
    },
  },
], []);

Return a File and it walks the ordinary parse path, header matching and all. Return an array of records and the SDK stages them without parsing. A throw becomes an error message on the upload step, and the person picks a file instead.

The bytes come through your own route, which reads the object and streams it back inside the session your app already issued. Nothing about AWS reaches the browser.

The result on submit

On submit, Updog Importer hands your handler every row grouped by source, each one carrying four independent flags.

import type { DataEditorResult, ResultRow } from "@updog/data-editor";

const CHUNK_SIZE = 2000;

const toRow = (entry: ResultRow<TrialRow>) => ({
  plotRef: entry.row.plotRef,
  site: entry.row.site,
  variety: entry.row.variety,
  harvestedOn: entry.row.harvestedOn,
  yieldTHa: Number(entry.row.yieldTHa),
  moisturePct: Number(entry.row.moisturePct),
  treatment: entry.row.treatment,
  notes: entry.row.notes ?? "",
});

const onComplete = useCallback(async (result: DataEditorResult<TrialRow>) => {
  const batchId = crypto.randomUUID();
  const rows = result.sources.flatMap((source) =>
    source.rows.filter((entry) => entry.isValid && !entry.isDeleted).map(toRow),
  );

  let part = 0;
  for (let start = 0; start < rows.length; start += CHUNK_SIZE) {
    part += 1;
    const answer = await fetch("/api/trials/parts", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        batchId,
        part,
        rows: rows.slice(start, start + CHUNK_SIZE),
      }),
    });
    if (!answer.ok) throw new Error((await answer.json()).message);
  }

  const closed = await fetch("/api/trials/complete", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ batchId, parts: part, rows: rows.length }),
  });
  if (!closed.ok) throw new Error((await closed.json()).message);

  await storeSynonyms(result.learnedSynonyms);
}, []);

Only valid rows travel, since a row S3 accepts is a row somebody has to find again later. The handler mints one batch id, posts the rows in chunks, then calls a second endpoint that closes the batch.

The chunk size is yours. S3 publishes no row count, and the numbers that bound it sit at the two ends. A single PUT carries up to 5 GB, AWS suggests multipart "when your object size reaches 100 MB", and your own endpoint accepts whatever body it accepts. Two thousand rows is a number chosen against that endpoint.

Throw when a route answers with a failure. Updog holds submit until your promise resolves, then empties the editor. A handler that traps the error and returns counts as a finished import, and the whole season clears the grid unwritten. A thrown error keeps the grid as it stands, with every mapping and hand correction on it.

The route that writes the object

The route holds the credentials, and the browser stops there.

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({});
const BUCKET = process.env.TRIALS_BUCKET;
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const SITE = /^[A-Z]{2}-[A-Z]{3}-\d{2}$/;

const FIELDS = ["plotRef", "variety", "harvestedOn", "yieldTHa",
  "moisturePct", "treatment", "notes"];
const HEADER = "plot_ref,variety,harvested_on,yield_t_ha," +
  "moisture_pct,treatment,notes";

const cell = (value) => '"' + String(value ?? "").replaceAll('"', '""') + '"';
const toCsv = (rows) => [HEADER]
  .concat(rows.map((row) => FIELDS.map((f) => cell(row[f])).join(",")))
  .join("\n") + "\n";

app.post("/api/trials/parts", async (request, response) => {
  const session = await getVerifiedSession(request);
  if (!session) return response.status(401).json({ message: "Not signed in" });

  const { batchId, part, rows } = request.body;
  if (!UUID.test(batchId)) {
    return response.status(400).json({ message: "Bad batch id" });
  }
  if (!Number.isInteger(part) || part < 1 || part > 9999) {
    return response.status(400).json({ message: "Bad part number" });
  }

  const bySite = new Map();
  for (const row of rows) {
    if (!SITE.test(row.site) || !session.sites.includes(row.site)) {
      return response.status(403).json({ message: "Unknown trial site" });
    }
    bySite.set(row.site, (bySite.get(row.site) ?? []).concat(row));
  }

  for (const [site, siteRows] of bySite) {
    const key = "harvest/season=" + session.season + "/site=" + site +
      "/batch=" + batchId + "/part-" + String(part).padStart(4, "0") + ".csv";
    try {
      await s3.send(new PutObjectCommand({
        Bucket: BUCKET,
        Key: key,
        Body: toCsv(siteRows),
        ContentType: "text/csv",
        ChecksumAlgorithm: "CRC32",
        IfNoneMatch: "*",
      }));
    } catch (error) {
      if (error.name !== "PreconditionFailed") {
        request.log.error({ key, name: error.name });
        return response.status(502).json({ message: describe(error) });
      }
    }
  }

  response.json({ part });
});

getVerifiedSession() stands in for your own server-side authentication check. The season comes off the session, and the site is checked against the codes that session is allowed to write. Every value that reaches the key is validated in the route, whatever the browser did first.

IfNoneMatch: "*" is what makes a repeat harmless. "If there's no existing object with the same key name in the bucket, the write operation succeeds, resulting in a 200 OK response. If there's an existing object, the write operation fails, resulting in a 412 Precondition Failed response." A part posted twice lands once. AWS also settles the race between two callers. "If multiple conditional writes or copies occur for the same object name, the first write operation to finish succeeds." There is no extra charge for any of it.

ChecksumAlgorithm asks S3 to verify what arrived against what the SDK says it sent, and CRC64NVME is the default when you name nothing. Objects are encrypted at rest with SSE-S3 unless you ask for something else.

A batch is readable once the manifest lands. That endpoint writes one small JSON object under _batches/, with the same If-None-Match, carrying the part count and the row count. Parts with no manifest behind them are an abandoned attempt, which is what a resubmitted import leaves, and a lifecycle rule can expire them.

The door the browser could open itself

S3 is the one destination in this cluster the browser could write to directly. A route can sign a PUT for one key and hand the URL over, and the upload skips your server entirely.

AWS describes what comes with that. "In essence, presigned URLs are bearer tokens that grant access to those who possess them." The URL works "multiple times, up to the expiration date and time", set as high as seven days through the SDKs. And a PUT to a key that exists replaces the object, since "Amazon S3 replaces the existing object with the uploaded object".

Keeping the write on the server buys one place where the rows are checked, one place where the key is built, and one place where the conditional header is set. For an import whose whole point is what lands, that is where it belongs.

When the write fails

A rejected write arrives as an error carrying name, and PreconditionFailed is the one to catch by hand, since it means the object is already there. $metadata.httpStatusCode holds the status behind it.

Throttling is handled for you. S3 answers 503 (Slow Down) while it scales a prefix to a higher rate, and the SDK classifies SlowDown as a throttling error and retries it with a longer base delay. The default is three attempts, one request and two retries, tuned through AWS_MAX_ATTEMPTS. The 2026 backoff timings need AWS_NEW_RETRIES_2026=true in the environment until they become the default.

Log the key and the error name, answer the browser with a sentence an agronomist can act on, and let the handler throw. All of it happens with the confirm dialog open and a spinner on the button, so anything on the result worth keeping gets copied inside the handler.

What reads it back

S3 stores the object. Amazon Athena reads it, and Athena has the rules S3 declined to have.

CREATE EXTERNAL TABLE trials.harvest (
  plot_ref     string,
  variety      string,
  harvested_on string,
  yield_t_ha   double,
  moisture_pct double,
  treatment    string,
  notes        string
)
PARTITIONED BY (season string, site string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
  'separatorChar' = ',',
  'quoteChar'     = '"',
  'escapeChar'    = '\\'
)
STORED AS TEXTFILE
LOCATION 's3://acme-trials/harvest/'
TBLPROPERTIES ('skip.header.line.count' = '1');

Three of those rules were settled in the browser hours earlier. The Open CSV SerDe needs to be told about a header row, which is what skip.header.line.count does. It "does not support embedded line breaks in CSV files", which is why the notes transformer collapses them. And it "Does not recognize empty or null values in columns defined as a numeric data type, leaving them as string", which is why yield and moisture are both required.

Dates keep their own rule. The SerDe reads a DATE column as days elapsed since 1 January 1970, so harvested_on is declared as a string. AWS names the way out. "To further convert columns to the desired type in a table, you can create a view over the table and use CAST to convert to the desired type." A column that can genuinely arrive empty takes the same road.

The site never appears in the file. It is a partition column, read out of the key the route built, which is the last thing that value does before it becomes a path.

The parts nobody ships for you

Updog Importer integrates with nobody. There is no S3 connector, no destination list, no webhook and no server of ours. onComplete hands your code an object, and the route, the key layout and the manifest in the middle are work you do.

S3 already ships its own ways in for the other case. The console takes a file up to 160 GB by drag and drop. aws s3api put-object sends one up to 5 GB in a single PUT, and aws s3 cp switches to multipart above 8 MB, which carries a single object to 50 TB across as many as 10,000 parts. AWS Transfer Family answers SFTP straight into the bucket. For a file your own team assembled, those are the shorter way in. Everything above exists for the export a research station sends, opened in a browser inside a session your app issued. Client-side and server-side CSV import names the jobs each of the two models fits.

The pieces you wrote and the next season

You wrote a schema with eight columns, one synonyms block, a key layout, a route that turns a chunk into objects under a conditional header, and a second endpoint that closes the batch. The file stays on the machine that opened it. The rows travel from your own front end to your own route and into your own bucket. The only party you added to the chain is yourself. Point the same setup at a React CSV importer modal or at the web component and the middle stays the same.

Next season the same station sends the same spelling of its own name, and a fourth one nobody has seen. The mappings are already stored, the new spelling costs one more line in synonyms, and the value that would have become a pound sign in an object key sits in a grid cell instead, in front of the person who typed it.