Experience
Home About Skills Projects Experience
Professional Experience
AI Agents & Multi-Agent Systems
-Designed LangChain-based multi-agent pipelines for task decomposition, coordination, and reasoning. Focused on agent orchestration for knowledge-intensive tasks.
- Experience with tokenization, embeddings, and transformer architecture (HuggingFace, PyTorch). Able to train and evaluate models for intelligent document processing and NLP automation tasks.
-
Local LLM Integration (LLaMA + Ollama)
- Deployed a fine-tuned version of LLaMA 3 using LoRA/QLoRA for specialization in documentation-based search tasks.
- The model is served locally via Ollama, exposing a simple HTTP API.
- Input prompt includes top-k document chunks retrieved from Qdrant + user query → model generates the final answer or summary.
- Training used a custom instruction dataset based on the ingested PDF corpus.
1. Database Change Data Capture (CDC) with Debezium
- Implemented Change Data Capture (CDC) pipelines using Debezium to capture real-time changes from relational databases such as Microsoft SQL Server, MySQL, Oracle, and PostgreSQL.
- Configured Debezium connectors to ensure minimal latency and high reliability in data streaming processes.
2. Data Streaming with Kafka
- Designed and managed Kafka topics to support high-throughput, real-time data ingestion.
- Ensured data consistency and fault tolerance through the implementation of robust Kafka configurations.
3. Data Processing and Loading
- Developed Python-based consumers for processing Kafka streams and loading enriched data into the Greenplum analytical database.
- Applied transformation and enrichment logic to prepare data for analytical use cases.
4. Microservices Architecture
- Designed and developed modular, scalable microservices for data ingestion, transformation, and delivery.
- Deployed microservices within a containerized environment using Docker.
- Orchestrated containers with Kubernetes to maintain high availability and scalability of data workflows.
5. Logging and Monitoring
- Centralized application logging using Logstash containers to enable real-time log tracking and diagnostics.
- Monitored system health and performance metrics via Prometheus, with dashboards and visualizations created in Grafana.
6. Data Analytics Pipeline
- Designed and optimized end-to-end pipelines capable of handling high-volume, real-time analytical workloads.
- Ensured seamless integration between source systems, streaming platforms, and the Greenplum analytical database.