The archive

1 posts in category “Data Platform”

Filtered by category: Data Platform ×

Tag

10x Engineer 1970 1970s Technology Emulation 2100 2100 Tech 3D Printing 3G 4000 BC 4G 4K 4K Monitoring 5G 8k AI AI Automation AI Integration AI Orchestrator AI Retraining AI Traffic Prediction AI anomaly detection AI consensus AI optimization AI orchestration AI sentiment analysis AI-Driven AI-Driven Debugging AI-powered fingerprinting AMD AWS AWS Lambda Accelerated Accelerated Computing Access Control Advanced Cryptography Agile Agile development Agility Air Traffic AirPods Alertmanager Ancient Artifacts Angular Ansible Apache Apache Beam Apache Flink Apache Jetty Apache Kafka Apache Pulsar Apollo Federation Apple Watch Architecture Argo Argo CD Argo Workflows ArgoCD Artificial Intelligence Asset Tracking Astronaut Asynchronous programming Audio Technology Audit Optimization Audits Augmented Reality Augmented reality contact lenses Authentication Automation Autonomous Vehicles Availability Azure Azure Functions BCP BERT BFD BGP BGP Routing BIND BLE BYOD Banking Banking Security Banking Tech Beer Beer Management Berlin Big Data Binary Tree Biochips Biohacking Bioinformatics BlackBerry Blackbox Blog Bluetooth Data Books Borg Bot Brain-Computer Interface Bring Your Own Device Bring Your Own Device (BYOD)Browser cache Burger Delivery BurgerOps Business Business Continuity Business Continuity Plan C-Level C-Level Communication CCIE CCNA CDC CI CI/CD CIFS CMDB CPU CPU Optimization CSS CUDA Cache Caching Optimization Camera Camera Integration Camera Protocol Camera Systems Cassandra CentOS Certificate Renewal Certificates Chaos Engineering ChatGPT Chatbot Checkpoint CloudGuard Checkpoint Gaia China Cinema Cisco AnyConnect ClickHouse Climate Control Climate Simulation Cloud Automation Cloud Computing Cloud Infrastructure Cloud Security Cloud Solutions Cloud-native Cloudflare Coffee CoffeeOps Communication Company Wiki Compiler Complex Architecture Complex Solutions Complex Systems Complex solution Compliance Computing Concurrency Concurrent Systems Consensus Algorithms Consistency Consul Container Orchestration Container Security Containerization Continuous Development Continuous Integration Cookies Cooling Systems Cross-Team Collaboration Cryptocurrency Cumulus Linux Custom Resource Definitions Cyber-Mechanical Integration Cyber-physical systems DDS DHCP DNA Computing DNS DNS Resolver DNS resolution Dark Matter Exploration Dark matter Dashboard Data Data Analytics Data Centers Data Extract Data Lake Data Logging Data Management Data Ownership Data Routing Data Science Data Streaming Data Transmission Data analysis Data orchestration Data security Database Replication Deadline Deadline Management Deep Learning Dell Dell OpenManage Dell Servers Deployment DevOps DevSecOps Developer Salary Developer Tools Device Telemetry Digital twin Disaster recovery Discord Discord Bots Distributed Consensus Distributed Ledger Distributed Real-Time Distributed Systems Distributed Teams Django Docker DockerHub Dogecoin Drives Drones DynamoDB E-Commerce ED25519 EDA ESXi ETL EVPN Ecosystem Edge Intelligence Electricity Electricity Optimization Elliptic curve cryptography Email Email Automation Energy Grid Optimization Energy Optimization Engineering Engineering Blog Engineering Design Engineering Solution Engineering innovation Enterprise Enterprise Solutions Envoy Ethereum Ethernet Event Streaming Event-Driven Architecture Event-driven programming Exoskeletons Explainable AI Explainable artificial intelligence Extract Extract, transform, and load (ETL)Extreme Programming F5 Load Balancer F5 Loadbalancer FPGA FTP FaaS Fabric Framework Fast UI Federated Systems Federated learning Fibre channel Fighter Jets Finance Firecracker Firecracker MicroVM Firewall Firmware Fitness Tracker Fitness Tracker Integration Fitness Trackers Flask Fleet Management Flink Flutter Football Analytics Fortinet Fortnite Framework Fries Frontend Engineering Fry Ordering Full-stack development Function as a Service GNMI GNU GNURooted GPS GPU GPU Acceleration GPUs GRPC Galaxy Game Boy GameBoy Advance Gameboy Gameboy Integration Gaming Security Gentoo Germany Git Integration GitHub GitLab Go GoPro Golang Google Cloud Functions Google Maps Google Maps API Google Speech API GraphDB Green IT Green technology Grok HR HSM HTTP HTTP Protocol HTTP/3 Hamburg Haptic technology Harbor Hardware Haskell Healthcare Hello World Helm High Performance Computing High Throughput High-energy physics Historical Data Home Automation Homegrown Alerts Homomorphic Encryption Hybrid Architecture Hybrid Computing Hybrid Systems Hyped Tech Hyper-V Hyperautomation Hyperledger Hyperledger Fabric Hyperloop IDE IDS IEEE IMAP IPS IPv6 ISMS IT ITIL Iceberg Icinga2 Industrial Industrial Automation Industrial Ethernet InfluxDB Infrastructure Infrastructure Monitoring Innovation Integration Testing Internet of Medical Things Internet of Things Interstellar Communication Intrusion Detection System Intrusion Prevention System Intrusion Prevention System (IPS)Inventory Management IoMT IoT IoT edge Istio JSON Jaeger JavaScript Jenkins Jinja2 Juniper Jurassic Park KPI KPI Monitoring Kafka Kanban Kindle Kindle Analytics Kindle Integration Kotlin Kubeflow Kubernetes LAMP LSTM Lambda Lambda Functions Latency Latency optimization Let's Encrypt Librenms Load Balancing Load ETL Load Testing Loadbalancing Logistics Technology Logstash London London Data Centers London tech Los Angeles Low Latency Lunar Infrastructure Lunch Ordering MCIV MMORPG MVC Mac OS X MacBook Machine Learning Mainframe Manufacturing MariaDB Mars Deployment Matrix Matrix Space MatrixSpace MediaPipe Memory Allocation Mesh Mesh Architecture Mesh Networks Messaging Metadata Metaverse Micro Data Centers Micro-VM Microsoft Microsoft Azure Microsoft Excel Microsoft Power Point Microsoft PowerPoint Microsoft Teams Microsoft Word Millisecond Latency Millisecond Precision Minecraft Minecraft Integration Minio Mission-Critical Systems Mobile App Development Mobile Payments Mobile payment Model-View-Controller Modular Money MongoDB Moon Multi Tenant Multi-Cloud Multi-Regional Deployment Multi-Tenancy Multi-Tier Architecture Multithreading Multiverse Mutual TLS NATS NFC NFT NGINX NLP NTP NVIDIA Nanoengineering NetBox Netflix Network Architecture Network Automation Network Infrastructure Network Management Network Monitoring Network Optimization Network Security Network Segmentation Network engineering Network function virtualisation Networking Neurofeedback Neuroinformatics Neuromorphic computing Nintendo Nintendo DS NixOS NoSQL Nobel Prize NumPy Nvidia GPUs OAuth 2.0 OLA OPA ORM ORM (Object-Relational Mapping)OSI model OSPF Observability Office Communication Office Logistics Office Tech Open Policy Agent Open Source Open Telemetry OpenSSL OpenShift OpenTelemetry Operational Efficiency Operational Level Agreement Optimization OracleDB Out of Band Outsourcing Overengineered Overengineered Solution Overengineered Solutions Overtime Optimization Ownership PKI Packer Pair programming Password Password Security Peer-to-Peer Networking Peer-to-Peer Networks Performance Performance Analysis Performance Optimization Petabyte Scale Petabyte Storage PhD Pipeline Plant Health Monitoring Plants Platform Architecture PlayStation Podman Pokémon Polymorphism Postgresql PowerDNS PowerPoint PowerPoint Presentation Printer Queue Private VLAN Profiler Progressive Web App Projects Proxmox PubSub Pulumi Python QR Code QRCode QUIC QUIC protocol Quality Control Quantum Quantum Cloud Quantum Computer Quantum Cryptography Quantum Key Distribution Quantum Mesh Network Quantum Networking Quantum Routing Quantum supremacy QuantumComputing ROS 2 RSA Encryption Rancher Raspberry Pi React Reactive Programming Reactive Systems Real-Time Real-Time Data Real-Time Data Processing Real-Time Logging Real-Time Processing Real-Time Synchronization Real-Time Systems Real-time Metrics Real-time Monitoring Recruiting Recursion Red Hat Enterprise Linux RedHat Enterprise Linux Redis Redundancy Regression Testing Reinforcement Learning Release Engineering Reliable Reliable Systems Remote Office Renewable Energy Requirement Management Requirements Resiliency Rest API Robotics Rocket Routing Routing Protocol Rust RxJS S3 SAP SCXML SDLC SDN SLA SMTP SNMP SPIFFE SQL SRE SSHFS SSL SaaS Salary Salary Analytics Samsung Satellite Data Satellite Operations Satellites Scalability Scalable Architecture Search Engine Secure Transport Security Self Hosting Self-driving Sentiment Analysis Serialization Server Optimization Server Orchestration Service Level Agreement Service Level Agreement (SLA)Service Workers Shipping ShitOps Site Reliability Engineering Site-2-Site Site-2-Site Connectivity Site-2-Site VPN Slack Smart Contracts Smart Home Smart Watches SmartHome Software development lifecycle Software-defined climate control Software-defined networking Software-defined networking (SDN)Solana Solar-Powered Solidity Sony Sound Simulation Space Space Communication Space Networking Space Tourism Spam Prevention Spam control Speech Recognition Speech-to-Text Sport Monitoring Sports Analytics Spotify Model Star Trek Star Wars State machine Stateless Architecture Steam API Storage Storage Solutions Streaming Strimzi Sustainability Sustainable Technology Swarm Intelligence Swarm Robotics Sway Switch Switch Architecture Switch Telemetry Synchronization System Architecture TCP TDD TFX TLS Encryption Tanks Tape Backup Tape Storage Task Management Team Event Team Events Tech Tech Company Tech Innovation Tech Innovations Tech Overengineering Tech Platforms Tech Solutions Tech Talks TechRadar Technical blog Technology Telegram Telemetry Television Television Monitoring Television Networks Television optimization Temperature Temporal Tendermint Tensor Processing TensorFlow TensorFlow Extended TensorFlow Lite Terabyte Data Terabyte Data Management Terabyte-scale Terrabyte Terraform Tesla Tesla Autopilot Testing ThinkPad Threema Time Sensitive Time-Sensitive Systems Tower Traefik Traffic Management Traffic Optimization Translator Translator Mesh Transportation Twitter Integration TypeScript UDP USA USB Unicorn Unit testing User experience VLAN VMware VMware NSX-T VMware Tanzu VMware Tanzu Kubernetes VPN VPN Security VR VXLAN Vegan Version Control Video Analysis Video Conferencing Video Logging Video Processing Virtual Assistants Virtual Machine Visualization Vue Vue.js WLAN WSL Waterfall Model Wayland Wearable technology Wearables Web Development Web Performance Web4 WebAssembly WebRTC WebSocket Website Optimization WhatsApp WiFi WiFi Optimization Wiki Windows 11 Windows 8 Windows Phone Windows Server Windows XP Wireshark WordPress Workshop World XAI XML (Extensible Markup Language)XMPP Xbox Zero-Trust adaptive filtering adaptive security appliance agile-methodology ai-ml airpods pro algorithm almalinux amazon amazon cloud ambient intelligence analytics android antivirus apache jetty integration api apple apple-maps arch linux arm-chip astronaut recruiting astronaut-communications audio streaming australia avengers backup bank bitcoin blazingly fast blockchain build or buy business intelligence cache-optimization capacity planning casio casio g-shock chat chatbots checkpoint ci-cd cilium cisco cisco firepower climate climate adaptation cloud cloud architecture cloud evangelist cloud storage cloudflare-workers complex complex pipeline complex pipelines configuration management container container runtime containers continuous delivery cooling cooling system cronjob cronjob automation cronjobs crypto cryptography cybersecurity cyborg data backup data encryption data engineering data extraction data pipeline data processing data storage data synchronization data warehouse database datacenter debugging decentralized decentralized systems deployment automation deserialization distributed distributed computing distributed proxy distributed system documentation dotnet e-books eBPF edge computing efficiency eks elasticsearch elk stack email delivery email processing encryption enterprise architecture enterprise integration enterprise service bus ephemeral etcd event-driven event-driven automation event-sourcing excel executive-dashboards fabric fastapi federated GraphQL federation fingerprinting fintech firewalling flowers football fortinet firewall frameworks fridge full-stack functional-programming future technology game of thrones gaming integration generative AI gesture-recognition git gitops gnu-hurd gobeyondmars grafana graph database graphql hackernews hadoop hashing headphones help-requests high availability homebrew homegrown https hundreds of servers hybrid cloud hybrid environment iPad iPad Fingerprinting iPad Integration iPhone immutable data immutable infrastructure immutable ledger incident-response infrastructure as code infrastructure management integrationtesting intelligent transportation systems inter-team communication internet explorer internet-tv internship interpreter java key performance indicator key performance indicator (kpi)kibana kvm lab lamp stack legacy systems lenovo linux loadbalancer lock management logging low code mTLS Certificates mac mini mainframe computing mars marvel marvel-api medical-gaming memory mesh network mesh networking mesh vpn message broker message-queue metallb microVM microkernel microservice microservices microservices architecture milliseconds mirrormaker mission-critical mobile mobile apps mobile computing mobile gaming modular architecture monitoring moon-infrastructure morale-mesh mqtt multi-cloud redundancy multi-cluster architecture multi-tier multiple teams multiteam collaboration music mysql natural language processing neo4j network network integration network technology neural networks nintendo wii nixops nmap no code noops notifications ocaml office automation office culture on-call online shopping opensearch orchestration overengineering overtime packet loss peer-to-peer petabyte petabyte data plant care platform platform as a service platform-engineering playlist polyglot ORM polyglot-persistence post-quantum power point presentation predictive-analytics printer printing project-management prometheus protocol proxy mesh qr-codes quantum blockchain quantum computing quantum encryption quantum entropy quantum-resistant cryptography raft consensus rclcpp real-time-analytics redhat reliability remote work request for comment requirements-management resilient responsive design rhel robot operating system robotic-exoskeletons router rsa rsync s3fs salary dashboard salary data san-francisco scalable scrum self-driving-cars server management serverless service mesh sftp signal orchestration signal processing single pane of glass smart desks smart fridge smart grid smart grids smart watch smartwatch sms software software architecture software deployment software development software testing solar power solid-state-drives solution space exploration space-technology spam spam-detection speed spotify ssh stateful stateless storage optimization streaming analytics switches tablets tanzu tape tape management team morale team-collaboration tech solution tech stack technical solution temperature control temperature monitoring terabyte test-driven development text-to-speech threat-detection tls traffic engineering transparency trend detection trpc turing award twitter twitter api uno video video integration video streaming virtual assistant virtual reality warsteiner web3 webshop website windows windows 10 workflow automation x11 xml

1 post

Serving Terrabyte-Scale S3 Telemetry Memory Queries with an Event-Sourced Mesh

Bartholomew Q. Overcast

At ShitOps we collect telemetry from every service, including detailed memory profiles from our serverless fleet. Those profiles land in S3 as immutable objects. Over the last year the archive …

Telemetry S3 Kafka graphql memory Temporal Terrabyte Flink ClickHouse Iceberg

Oct 3, 2026 5 min read Read