Skip to content

Conversation

@NirYanay2005
Copy link

Summary

NIFI-15568

This change improves Apache Iceberg integration in NiFi by addressing two related issues:

  1. Adds support for configuring the S3 storage class in S3IcebergFileIOProvider, which is required for certain on-prem or S3-compatible object stores.
  2. Fixes timestamp type compatibility issues when writing Parquet-backed Iceberg tables by converting java.sql.Timestamp values to java.time.LocalDateTime, including correct handling for nested records, collections, and partitioned tables.

The timestamp fix ensures compatibility with Iceberg’s internal expectations and allows writes to succeed when timestamp columns are used as partition keys.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-15568
  • Pull Request commit message starts with Apache NiFi Jira issue number, such as NIFI-15568
  • Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using ./mvnw clean install -P contrib-check
    • JDK 21
    • [] JDK 25

Licensing

  • New dependencies are compatible with the Apache License 2.0
  • New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

Copy link
Contributor

@exceptionfactory exceptionfactory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for submitted the revised pull request @NirYanay2005.

Please review the pull request instructions and run a build with the contrib-check profile enabled to correct formatting issues such as missing license headers.

@exceptionfactory
Copy link
Contributor

Please note that each commit must also be signed with a key associated with your GitHub profile for verification.

@NirYanay2005
Copy link
Author

I added a key to my commits and fixed all contrib-check problems.

@exceptionfactory
Copy link
Contributor

I added a key to my commits and fixed all contrib-check problems.

Thanks @NirYanay2005, the initial commit is now signed, but the subsequent commits are not. Can you squash all commits and this branch to ensure they are all signed?

Removed unnecessary code that already existed in main

Added licenses and formatting

Fixed all contrib-check issues
@NirYanay2005
Copy link
Author

I added a key to my commits and fixed all contrib-check problems.

Thanks @NirYanay2005, the initial commit is now signed, but the subsequent commits are not. Can you squash all commits and this branch to ensure they are all signed?

Ok, I squashed all commits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants