97 things every data engineer should know : collective wisdom from the experts

Includes index.

Includes bibliographical references and index.

A (book) case for eventual consistency / Denise Koessler Gosnell, PhD -- A/B and how to be / Sonia Mehta -- About the storage layer / Julien Le Dem -- Analytics as the secret glue for microservice architectures / Elias Nema -- Automate your infrastructure / Christiano Anderson -- Automate your pipeline tests / Tom White -- Be intentional about the batching model in your data pipelines / Raghotham Murthy -- Beware of silver-bullet syndrome / Thomas Nield -- Building a career as a data engineer / Vijay Kiran -- Business dashboards for data pipelines / Valliappa (Lak) Lakshmanan -- Caution : data science projects can turn into the emperor's new clothes / Shweta Katre -- Change data capture / Raghotham Murthy -- Column names as contracts / Emily Riederer -- Consensual, privacy-aware data collection / Katharine Jarmul -- Cultivate good working relationships with data consumers / Ido Shlomo -- Data engineering !=Spark / Jesse Anderson -- Data engineering for autonomy and rapid innovation / Jeff Magnusson -- Data engineering from a data scientist's perspective / Bill Franks -- Data pipeline design patterns for reusability and extensibility / Mukul Sood -- Data quality for data engineers / Katharine Jarmul -- Data security for data engineers / Katharine Jarmul -- Data validation is more than summary statistics / Emily Riederer -- Data warehouses are the past, present, and future / James Densmore -- Defining and managing messages in log-centric architectures / Boris Lublinsky -- Demystify the source and illuminate the data pipeline / Meghan Kwartler -- Develop communities, not just code / Emily Riederer -- Effective data engineering in the cloud world / Dipti Borkar -- Embracing data silos / Bin Fan and Amelia Wong -- Engineering reproducible data science projects / Dr. Tianhui Michael Li -- Five best practices for stable data processing / Christian Lauer -- Focus on maintainability and break up those ETL tasks / Chris Moradi

Take advantage of today's sky-high demand for data engineers. With this in-depth book, current and aspiring engineers will learn powerful real-world best practices for managing data big and small. Contributors from notable companies including Twitter, Google, Stitch Fix, Microsoft, Capital One, and LinkedIn share their experiences and lessons learned for overcoming a variety of specific and often nagging challenges. Edited by Tobias Macey, host of the popular "Data engineering podcast", this book presents 97 concise and useful tips for cleaning, prepping, wrangling, storing, processing, and ingesting data. Data engineers, data architects, data team managers, data scientists, machine learning engineers, and software engineers will greatly benefit from the wisdom and experience of their peers.--