Log Analyzer — PySpark
Build PySpark job analyzing web server logs for insights.
PySparkLogsAnalytics
Mega Prompt — Copy & Paste into AI
You are a world-class big data engineer with 20+ years of experience shipping production software used by millions. I want you to build: A Web Server Log Analyzer using PySpark INPUT: Apache/Nginx access log files (GB-scale) TRANSFORMATIONS: - Parse log lines (regex): IP, timestamp, method, URL, status, size, user_agent - Filter bot traffic (user_agent patterns) - Aggregate: requests per hour, top 10 URLs, status code distribution, top IPs, avg response size - Sessionize: group requests by IP with 30-min timeout - Detect anomalies: IPs with >1000 requests (potential attack) OUTPUT: Parquet files for each analysis, summary JSON INCLUDE: complete PySpark script, sample log data generator, run instructions Before you start coding, ASK ME: 1. What should the PROJECT NAME be? 2. Any features you want to ADD or REMOVE from the list above? 3. Any specific DESIGN PREFERENCES? (Colors, style, dark/light?) 4. Any specific TECH CHANGES? (Different database, state management, etc.?) After I answer, present a brief PROJECT PLAN for my approval. Then generate COMPLETE, RUNNABLE code for every file. No placeholders. No TODO comments. Production-ready.
How to use this prompt
Copy the prompt above → open ChatGPT, Claude, Gemini, Copilot or DeepSeek → paste → the AI will interview you about your project, then build complete production-ready code.
Works with:
ChatGPT-5
Claude 4.5
Gemini 2.5 Pro
Copilot
DeepSeek V3
Mistral
About this Data Engineering Prompt
The Log Analyzer — PySpark is a battle-tested mega prompt designed for the Apache Spark stack. Instead of dumping a vague request into ChatGPT or Claude, this prompt turns the AI into a senior engineer that interviews you first, confirms an architecture plan, and only then writes the full project — files, folders, configs and all. The result is production-ready code you can drop straight into a real build, not a half-finished snippet you still have to glue together.
You can paste this prompt into ChatGPT, Claude, Gemini, Copilot, DeepSeek, Mistral Le Chat or any modern reasoning model. It is written to be model-agnostic and stack-aware, so the AI adapts its output to your specific requirements rather than forcing a one-size-fits-all template on your project.
What you'll get
- PySpark
- Logs
- Analytics
How to Use the Log Analyzer — PySpark
- Copy the full prompt from the box above using the "Copy Prompt" button. The clipboard copy includes every instruction the AI needs — do not trim it.
- Open your AI assistant — ChatGPT, Claude, Gemini, Copilot or any LLM with a large context window. Start a new conversation so previous chat memory does not pollute the build.
- Paste the prompt and hit send. The AI will ask you a structured set of discovery questions about your Apache Spark project — features, screens, auth, payments, design style and target users.
- Answer in plain English. You do not need to know technical jargon. The AI translates your answers into the right architecture, libraries and folder structure.
- Confirm the plan. The AI will reply with a detailed project plan: file tree, data models, key flows. Approve it or ask for changes before code generation starts.
- Receive complete code. Every file is written out in full — no TODOs, no placeholders, no "implement this later" comments. Drop it into your IDE and run.
Why this Prompt Outperforms a Plain Request
Most developers write prompts like "build me a data engineering app" and get back generic, half-broken boilerplate. The Log Analyzer — PySpark works better for three concrete reasons:
- Role priming — the AI is told to act as a senior engineer with 20+ years of production experience in Apache Spark, raising its quality baseline.
- Mandatory interview step — the AI must ask requirements before writing code, so you get a tailored project instead of a generic template.
- Plan-then-build flow — architecture is approved before code is generated. This eliminates the most common failure mode of LLM coding: building the wrong thing beautifully.
- Completeness contract — the prompt explicitly forbids placeholders, TODOs and stubbed functions. Every file you receive is meant to compile and run.
Frequently Asked Questions
Is the Log Analyzer — PySpark free to use?
Yes. Every prompt on AI Prompts Lib is 100% free, no signup or paywall. Copy it, paste into your AI of choice, and start building. We do not log your inputs or your generated code.
Which AI models work best with this prompt?
This prompt is tuned for any modern large-context model: Claude 4 / 4.5 Sonnet, ChatGPT-5 / GPT-4 Turbo, Gemini 2.5 Pro, DeepSeek V3 and Copilot Chat. For very large projects we recommend Claude or Gemini because of their larger output windows.
Can I modify the prompt for my own use case?
Absolutely. Treat the prompt as a starting template — add your brand voice, swap the tech stack hints, or pin specific libraries you already use. The interview + plan + build structure is the magic, not the exact wording.
What if the AI's output has bugs?
Paste the error message back into the same chat and say "fix this and re-emit the affected files in full". Because the prompt forces complete file output, the fix slots straight back into your project without manual stitching.