Troubleshooting

This document provides guidance for diagnosing and resolving common issues encountered during the development and operation of the IT Ticketing Service. It is intended for developers responsible for maintaining and extending the application.

Application Startup Issues

Problems that prevent the application from starting successfully are often related to external dependencies or local environment configuration.

Database Connection Failures

The application will fail to start if it cannot establish a connection with the MySQL database.

Symptoms:

• Log messages containing HikariPool-1 - Exception during pool initialization or Communications link failure.
• Application context fails to load, and the process exits.

Troubleshooting Steps:

1. Verify Configuration: Check the database connection properties in src/main/resources/application.properties.

   # src/main/resources/application.properties
   
   # Ensure the URL, username, and password are correct for your environment.
   spring.datasource.url=jdbc:mysql://legacydb.next26.slalomlab.com:3306/ticketing?createDatabaseIfNotExist=true
   spring.datasource.username=ticketing_user
   spring.datasource.password=changeme

2. Check Network Connectivity: The default configuration points to a remote database (legacydb.next26.slalomlab.com).

   • Ensure you have network access to this host and port (3306).
   • Check for firewall rules or VPN requirements that might be blocking the connection.
   • Use a tool like telnet or nc to test connectivity: telnet legacydb.next26.slalomlab.com 3306.

3. Local Docker Environment: If you are running a local MySQL instance via the provided docker-compose.yml, ensure the container is running and healthy.

   • Start the services if not already running: docker-compose up -d.
   • Run docker ps or docker-compose ps to verify the ticketing-mysql container is Up and healthy.
   • Check container logs for errors: docker logs ticketing-mysql.
   • If using the local container, update spring.datasource.url to jdbc:mysql://localhost:3306/ticketing.

For more details on configuration, see the Configuration Guide.

RabbitMQ Connection Errors

A failure to connect to the RabbitMQ broker will also prevent the application from starting.

Symptoms:

• Log messages containing AmqpConnectException or Broker not available.
• Endless reconnection attempts printed to the console.

Troubleshooting Steps:

1. Verify Configuration: Check the RabbitMQ properties in application.properties. The default configuration expects a local instance.

   # src/main/resources/application.properties
   spring.rabbitmq.host=localhost
   spring.rabbitmq.port=5672
   spring.rabbitmq.username=guest
   spring.rabbitmq.password=guest

2. Check RabbitMQ Service: Ensure the RabbitMQ service is running. If using the provided docker-compose.yml:

   • Start the services if not already running: docker-compose up -d.
   • Run docker ps or docker-compose ps to verify the ticketing-rabbitmq container is Up and healthy.
   • Check the container's health status and logs: docker logs ticketing-rabbitmq.
   • Access the RabbitMQ Management UI at http://localhost:15672 (credentials: guest/guest) to verify the broker is operational.

Port Conflicts

The application will fail to start if its configured server port is already in use.

Symptoms:

• Log messages containing Port 8080 was already in use or Web server failed to start.

Troubleshooting Steps:

1. Identify the Conflicting Process: Use system tools to find which process is using port 8080.

   • Linux/macOS: sudo lsof -i :8080
   • Windows: netstat -ano | findstr :8080

2. Resolve the Conflict:

   • Stop the other process if it is not needed.
   • Alternatively, change the application's port in application.properties by modifying the server.port property.

   # Change to an available port, e.g., 8081
   server.port=8081

Configuration Errors

Malformed or missing properties can cause startup failures.

Symptoms:

• Could not resolve placeholder errors if a property is referenced but not defined.
• InvalidConfigurationPropertyNameException for illegal characters in property names.

Troubleshooting Steps:

• Carefully review application.properties for typos or syntax errors.
• Ensure all necessary properties for the active Spring profiles are present.
• Refer to the Configuration Guide for a complete list of required properties.

Runtime Issues

These issues occur while the application is running, typically during API requests or background processing.

API Errors and Debugging

When an API call fails, the HTTP status code and response body provide initial clues.

• 400 Bad Request: Often caused by validation failures. The response body will typically contain details about which fields are invalid. Ensure the request payload matches the API contract and validation rules on the DTOs.
• 404 Not Found: The requested resource (e.g., /api/tickets/{id}) does not exist. Verify the ID is correct.
• 500 Internal Server Error: A generic server-side error. This requires inspecting the application logs for a stack trace to identify the root cause.

To get more detailed request information, increase the logging level for the Spring Web module:

# src/main/resources/application.properties
# Change from INFO to DEBUG or TRACE for detailed request/response logging
logging.level.org.springframework.web=DEBUG

For a detailed breakdown of API error codes, see the API Error Handling Guide.

Transaction Failures

The application uses Spring Data JPA, which manages transactions automatically. Failures can still occur.

Symptoms:

• TransactionSystemException, often wrapping a RollbackException.
• DataIntegrityViolationException when database constraints (e.g., unique keys) are violated.

Troubleshooting Steps:

1. Enable SQL Logging: To see the exact SQL statements being executed by Hibernate, enable show-sql.

   # src/main/resources/application.properties
   spring.jpa.show-sql=true
   spring.jpa.properties.hibernate.format_sql=true # Makes the SQL more readable

2. Analyze the Cause: Inspect the logs for the underlying exception.

   • A ConstraintViolationException points to a conflict with the database schema. Check your entity data against existing records.
   • Deadlocks or other database-level errors will be reported by the JDBC driver.

Message Publishing Problems

Failures to publish messages to RabbitMQ can occur if the connection is lost or the message is invalid.

Symptoms:

• AmqpException in the logs.
• Expected side effects of message consumption (e.g., downstream processing) do not occur.

Troubleshooting Steps:

1. Check RabbitMQ Connection: Use the /actuator/health endpoint to verify the rabbit component is UP.

2. Increase AMQP Logging: Get more verbose output from the RabbitMQ client library.

   # src/main/resources/application.properties
   logging.level.org.springframework.amqp=DEBUG

3. Check RabbitMQ Management UI: At http://localhost:15672, verify that the target exchange (ticket.exchange) exists and that there are no connection errors.

Database Troubleshooting

Issues related to database performance, schema, and data integrity.

Connection Pool Exhaustion

If all connections in the pool are in use and requests for new connections time out, the application will become unresponsive.

Symptoms:

• SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out.
• Slow API response times, eventually leading to timeouts.

Troubleshooting Steps:

1. Monitor the Connection Pool: Use the /actuator/health endpoint, which provides basic information. For deeper metrics, expose the /actuator/metrics endpoint and inspect hikaricp.* metrics.
2. Investigate Long-Running Queries: A common cause is one or more slow queries holding connections for too long. Use the query performance troubleshooting steps below.
3. Tune Pool Size: If high throughput is legitimate, you may need to increase the maximum pool size. This should be done cautiously in coordination with the database's maximum allowed connections.

Query Performance Issues

Slow queries can degrade the performance of the entire application.

Troubleshooting Steps:

1. Identify Slow Queries: Enable SQL logging (spring.jpa.show-sql=true) in a development environment to observe the queries generated by JPA.
2. Analyze Query Plans: Copy the slow query from the logs and run it directly against the database using EXPLAIN. The output will show if the query is using appropriate indexes.
3. Add Indexes: If the query plan indicates full table scans on large tables, add indexes to the columns used in WHERE clauses and JOIN conditions.
4. Optimize JPA Queries: Review your Spring Data repository methods. Complex method names can sometimes generate inefficient queries. For complex logic, consider using a custom query with @Query.

Schema Mismatch Errors

The application uses spring.jpa.hibernate.ddl-auto=update, which attempts to automatically update the schema based on JPA entity definitions. This is not safe for production and can lead to issues.

Symptoms:

• SchemaManagementException on startup.
• Runtime errors indicating a missing column or table.

Troubleshooting Steps:

• Disable ddl-auto in Production: For production environments, set spring.jpa.hibernate.ddl-auto=validate or none and use a dedicated database migration tool like Flyway or Liquibase to manage schema changes reliably.
• Compare Entity and Schema: Manually compare the fields and annotations in your @Entity classes with the table structure in the database to find discrepancies.

Messaging Troubleshooting

Diagnosing problems with RabbitMQ message publishing and broker configuration.

Message Not Published

Symptoms:

• An API call succeeds (e.g., creating a ticket), but no message appears in the queue.
• No errors are visible in the application logs.

Troubleshooting Steps:

1. Verify Code Logic: Use a debugger to step through the code path that publishes the message. Ensure the RabbitTemplate.convertAndSend() method is being called as expected.
2. Check RabbitMQ Management UI: Go to http://localhost:15672.
   • Select the ticket.exchange.
   • Check if there are bindings to it. A message published to an exchange with no bindings is silently dropped.
   • The rabbitmq.queue.name (ticket.queue) should be bound to the exchange with the rabbitmq.routing.key (ticket.routing.key).

Queue/Exchange Not Found

Symptoms:

• An AmqpException in the logs indicating the channel was closed because the exchange or queue does not exist.

Troubleshooting Steps:

1. Verify Broker State: Use the RabbitMQ Management UI to confirm that the exchange (ticket.exchange) and queue (ticket.queue) defined in application.properties actually exist.
2. Check Declarations: Spring AMQP can be configured to automatically declare exchanges, queues, and bindings. Ensure this configuration is present and correct. If it's missing, the components must be created manually on the broker.

Message Serialization Errors

Symptoms:

• MessageConversionException or a Jackson JsonProcessingException in the logs when attempting to publish a message.

Troubleshooting Steps:

• Ensure the object being sent as a message payload is properly serializable to JSON.
• Check that the object has a default constructor and getters for all serialized fields.
• Look for problematic field types (e.g., complex objects without custom serializers) that Jackson cannot handle by default.

Diagnostic Tools

Leverage built-in and external tools to diagnose issues effectively.

Application Logs Analysis

The primary source of information. Key loggers to configure in application.properties:

# Set to DEBUG or TRACE for more detail
logging.level.com.slalom.demo.ticketing=DEBUG # Application-specific code
logging.level.org.springframework.web=INFO    # For HTTP requests/responses
logging.level.org.springframework.amqp=INFO   # For RabbitMQ operations
logging.level.org.hibernate.SQL=DEBUG         # Alternative to show-sql
logging.level.org.hibernate.type.descriptor.sql=TRACE # To see bind parameter values

Database Query Logging

The most direct way to see the SQL generated by Hibernate.

# src/main/resources/application.properties
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.format_sql=true

RabbitMQ Management Interface

An indispensable web-based tool for inspecting the state of the RabbitMQ broker.

• URL: http://localhost:15672
• Default Credentials: guest / guest
• Use it to:
  • View active connections and channels.
  • Inspect exchanges, queues, and their bindings.
  • Monitor message rates and see queued messages.
  • Manually publish messages for testing consumers.

Actuator Diagnostic Endpoints

Spring Boot Actuator provides production-ready endpoints for monitoring.

• Health Check: GET /actuator/health
  • Provides a consolidated health status of the application and its connections to the database and RabbitMQ. The show-details=always property ensures detailed information is included.
• Application Info: GET /actuator/info
  • Displays general application information.

For more on Actuator, see the Monitoring Guide.

Getting Help

When you cannot resolve an issue on your own, provide clear and complete information to the team.

Log Collection for Support

Before reporting an issue, collect the following artifacts:

1. Full Application Logs: Capture the complete log output from application startup until the error occurs.
2. Relevant Configuration: Provide the contents of application.properties (redact any sensitive passwords).
3. Actuator Health Output: The JSON output from the /actuator/health endpoint.
4. Steps to Reproduce: A precise description of the actions taken that led to the error.

Issue Reporting Guidelines

Create a ticket or issue with the following structure:

• Summary: A concise title describing the problem (e.g., "API returns 500 error when creating a ticket with a null priority").
• Steps to Reproduce:
  1. Start the application with mvn spring-boot:run.
  2. Send a POST request to /api/tickets with the following payload: {...}.
  3. ...
• Expected Behavior: A clear description of what you expected to happen.
• Actual Behavior: A description of what occurred, including the full error message and stack trace from the logs.
• Environment:
  • Application Version/Git Commit:
  • Java Version:
  • Operating System:
  • Dependencies: (e.g., Local Docker, Remote Staging DB)