Friday 11:30 a.m.–noon
Creating Bomb-Proof Data Importers
Daniel Lindsley
- Audience level:
- Intermediate
- Category:
- Best Practices & Patterns
Description
Building importers that consume other peoples' data can sometimes feel a bit like dumpster-diving. Non-compliant formats, network errors and missing data are just some of the many ways your importer is a ticking time bomb. This talk presents ways to diffuse that bomb, so when the data's broke, you can still be the hero.
Abstract
Building importers that consume other peoples' data can sometimes feel a bit like dumpster-diving. Non-compliant formats, network errors and missing data are just some of the many ways your importer is a ticking time bomb. This talk presents ways to diffuse that bomb, so when the data's broke, you can still be the hero.
To help preserve your hero status, we'll be talking about:
* Common pitfalls
* How to structure your importer better
* The importance of logging & how to manage it
* Failure cases & how to address them
* Processing the data
We'll also talk about what tools you can use to help get the job done, how to keep it running fast/light/smooth & a few war-stories of my mistakes laid bare for some good-old public shaming.