Skip to main content

· 2 min read

Flight Landing Log​

Objective​

Deploy Filenet on customer-owned AWS EKS infrastructure for use in growth opportunity to transition to AWS Gov Cloud, and provide customer with hands-on experience and education on Kubernetes and deploying containerized applications on AWS

Milestones​

Completed​

  • Determine Deployment Size based on Filenet PoC Hardware Requirements
  • Provision AWS Cluster
  • Prepare AWS account
    • Install command line tools
  • Stage FileNet images
  • Deploy EKS Cluster
  • Install OpenLDAP
  • Deploy the Database
  • Deploy the FNCM Operator
  • Enable and Demonstrate Content Search Services
  • Educate customer on Kubernetes and AWS

Decisions and Action Items (DAI)​

  • Optional RDS Configuration
    • Process Documented, customer may proceed as needed
  • Discuss and demonstrate Content Replication
    • Documented and Handed-off to CSM team
  • Walk through GraphQL use cases
    • Documented and Handed-off to CSM team
  • Customer load-balancer decision
    • Load-balancer and PKI configuration to be determined by the customer
    • Affected components:
      • Standalone Process Designer Setup
      • Creation of Workflows
  • IBM Enterprise Records Support
    • Tickets were raised with the product team, long-term enhancement request to support IER on AWS
    • Handed-off to CSM team to follow-up

Lessons learned​

  • Namespace Deletion: Stuck in Terminating State
    • Proceeded with Manual Deletion Documentation to remove the finalizer
    • This IBM Documentation pertains to IBM Cloud Private but is applicable to all clusters including AWS
  • Documentation Feedback
    • Documentation to be more generalized, clear and applicable to other customers
    • ECR Tag directory paths, folder structure tags unavailable
  • Lack of IER Support
    • Not officially supported on AWS by IBM

· One min read

Date​

Flight Logs contain information relating to steps completed between 03/19/24 - 03/21/24

Key Accomplishments​

Work In Progress​

  • Working with support team to resolve IER deployment failure.

Challenges​

  • IER Image: Client is getting Back off error message for IER pod.

Action Items​

  • Track the case TS015679454 and resolve the issue.
  • Look into s3 config without access keys - enhancement request was approved by product team last year. It improves content replication setup.

Next Steps​

  • Client to run through the steps to Install NGINX Controller.
  • Look into the certificate setup and recommend next steps, including any other fields in the YAML that have to be filled.
  • Check if cert is now valid from ICN, etc. If so, reimport cert into keystore and retry launching PD.
  • Continuing to track the IER image pull issue support case. Deploy IBM Enterprise Record and TM pod setup for IER.

Tracking​

Cases open: 1

  • Case TS015679454

· 2 min read

Date​

Flight Logs contain information relating to steps completed between 03/12/24 - 03/14/24

Key Accomplishments​

  • Client is setting up their AWS gov cloud environment in parallel. Overall the steps in this solution guide worked well for AWS gov cloud.
  • Guided client on how to add css index area for another object store. In the css sections under 'initialize_configuration'need index area per OS.

Work In Progress​

  • Walked through running the commands to check which file storage the index areas map to. (eg: how /opt/ibm/indexareas maps to efs)
  • Went through cert manager setup.
  • Working with support team to resolve IER deployment failure.

Challenges​

  • Ceritficate issue: PD - Security configuration, says not secure and certificate not valid on all the urls. Took a closer look at cert setup- matched tls secret name with the private secret key ref. Cert still did not come up after doing this. FQDN is not filled out, discussed options.

  • IER Image: Client is getting Back off error message for IER pod.

Action Items​

  • Track the case TS015679454 and resolve the issue.
  • Look into s3 config without access keys - enhancement request was approved by product team last year. It improves content replication setup.

Next Steps​

  • Look into the certificate setup and recommend next steps, including any other fields in the YAML that have to be filled.
  • Check if cert is now valid from ICN, etc. If so, reimport cert into keystore and retry launching PD.
  • Continuing to track the IER image pull issue support case. Deploy IBM Enterprise Record and TM pod setup for IER.

Tracking​

Cases open: 1

  • Case TS015679454

· One min read

Date​

Flight Logs contain information relating to steps completed between 03/05/24 - 03/07/24

Key Accomplishments​

  • Client was able to apply FNCM CR. Pods came up without errors and they were able to access ACCE UI.

Work In Progress​

  • IER deployment failed. New case has been opened to look into the issue before next weeks session.

Challenges​

  • EKS Cluster clean up : Client environment has a clean up process that deletes EKS cluster to save cost. This process ran again and client has to provision cluster again. EKS cluster was deleted as part of their monthly clean up cycle.

  • IER Image: Client is getting Back off error message for IER pod.

Action Items​

  • Track the case TS015679454 and resolve the issue before next weeks sessions.

Next Steps​

  • Deploy IBM Enterprise Record.

Tracking​

Cases open: 1

  • Case TS015679454

Cases closed: 1

  • Case TS015569690

· One min read

Date​

Flight Logs contain information relating to steps completed between 02/27/24 - 02/29/24

Work In Progress​

  • Configure CR and resolve issues that we encounter while applying the yaml.

Challenges​

  • DBA Keytool: We were getting image pull back error for “dba-keytool-initcontainer:23.0.2” while applying the CRD and pods are in crash back loop.

Action Items​

Next Steps​

  • Client to test the solution for DBA Keytool image issue.
  • Go through the FNCM CRD and fill out the configuration.

Tracking​

Cases open: 1

  • Case TS015569690

· 2 min read

Date​

Flight Logs contain information relating to steps completed between 02/20/24 - 02/22/24

Key Accomplishments​

  • Client scripted certain steps which should help if EKS cluster has to be reprovisioned and we need to redo the installation.

Challenges​

  • EKS Cluster clean up : Client environment has a clean up process that deletes EKS cluster to save cost. Some of the resources in EKS failed to delete during this process. Client had to spend most of the week to create a new cluster and run through what we did the previous week.

Lessons Learned​

  • Try to script/automate most of the steps to speed up the process if we run into issues like cluster deletion
  • Database connection over SSL in CRD is disabled since we follow a MVP mode. In a production deployment client can set it up for more security.

Action Items​

  • Client wants to know how to move their environment specific data into the extracted case package in an upgrade scenario.
  • Any best practices and guidelines we can recommend on moving secrets to aws, etc in a production setting for security purposes.
  • Research egress part in CR. For the time being we kept that section but set the sc_restricted_internet_access: false
  • Documented differences in CR in the new case file.

Next Steps​

  • Go through the FNCM CRD and fill out the configuration.

Tracking​

· 2 min read

Date​

Flight Logs contain information relating to steps completed between 02/13/24 - 02/15/24

Key Accomplishments​

  • Client provisioned a small EKS cluster and was able to go through staging steps.
  • Customer was able to do the workaround provided yesterday to support hierarchical image naming convention in Amazon ECR.
  • Deployed OpenLDAP & Postgresql.
  • Successfully deployed the FileNet Operator into the customer's AWS EKS environment by manual process.

Challenges​

  • Postgresql Failure: Permission issue while deploying postgresql.

    • Error Message: "chmod: changing permission of '/var/lib/postgresql': Operation not permitted"
    • Solution: Client used “efs-sc” (file storage) from the the FNCM Operator CR yaml. The postgres-pvc.yaml said to use “ebs-gp3-sc” (block storage). Switched the storageclass to fix the error.

  • Amazon ECR: ECR has a flat structure so to maintain hierarchical naming structure of Filenet images client has to create separate repositories within ECR.

Lessons Learned​

  • Make a note in the documentation to make it clear to use EBS(Block storage) instead of EFS for Postgresql.
  • Amazon ECR behavior and workaround needs updated in the documentation.
  • Both load balancer and nginx are needed. Nginx only works on ports 80 and 443 and other port goes through the load balancer.

Action Items​

Next Steps​

  • Go through the FNCM CRD and fill out the configuration.

Tracking​

· One min read

Work in Progress​

  • The team scaled the FNCM operator down to 0 and set readOnlyRootFilesystem to false in the IER deployment directly.
  • The team then added explicit Dynatrace annotations to the IER deployment that were supposed to disable the agent injection.
  • We then modified the daemonset for Dynatrace to scale it to 0. After applying this change, the IER pods came online.

Currently Tracking​

  • The client will verify the IER pod comes up again after scaling the FNCM operator back up.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014753369

Cases closed: 3

  • Case TS014232963
  • Case TS014348824
  • Case TS014370797

· One min read

Work in Progress​

  • The team continued working on the IER deployment.
  • The IER pods were in a failed state, but after further investigation it appears DynaTrace is causing the issue.
  • The team collected logs from this session and opened a case on the issue.
  • The team then walked through some Workflow functionality.

Currently Tracking​

  • The team has opened a case and will be working closely with IBM support to resolve this issue.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014753369

Cases closed: 3

  • Case TS014232963
  • Case TS014348824
  • Case TS014370797

· One min read

Work in Progress​

  • The team checked the status of the IER deployment and noticed the IER pod was failing.
  • The team spent time troubleshooting the IER deployment.
  • The team tried adjusting the FileNet CR and gathered logs from the session.

Currently Tracking​

  • The team will examine the logs gathered from this session for further troubleshooting.

Next Steps​

Tracking​

Cases open: 0

Cases closed: 3

  • Case TS014232963
  • Case TS014348824
  • Case TS014370797

· One min read

Work in Progress​

  • With the help of IBM support the team troubleshooted some workflow errors.
  • There were some settings misconfigured in the workflow that were causing errors.
  • The team gathered logs from today’s session and will be working closely with IBM support to investigate the workflow error.
  • The team configured the CR to enable IER.

Currently Tracking​

  • The client's team will apply the CR changes and monitor its status.
  • We will continue to work closely with IBM support and have provided them logs of today’s session.

Next Steps​

Tracking​

Cases open: 0

Cases closed: 3

  • Case TS014232963
  • Case TS014348824
  • Case TS014370797

· One min read

Work in Progress​

  • With the help of IBM support the team continued to troubleshoot the FileNet Navigator issue.
  • The team discovered that Navigator was trying to connect using the wrong port. The team adjusted the port within CR and was able to access Navigator.
  • The team walked through using workflows within Process Designer.
  • The team discussed upcoming use cases now that the major errors have been addressed.

Currently Tracking​

  • The team will be prepping for upcoming use cases and will be preparing documentation on GitHub.

Next Steps​

Tracking​

Cases open: 0

Cases closed: 3

  • Case TS014232963
  • Case TS014348824
  • Case TS014370797

· One min read

Work in Progress​

  • Today the team brainstormed and discussed possible solutions for the Navigator Error.
  • The team walked through the whiteboard and discussed remaining tasks on the FileNet project.

Currently Tracking​

  • The team is continuing to work closely with IBM support on this issue.
  • The team is actively working with IBM support to get an expert from the Navigator team to join a future session.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014370797

Cases closed: 2

  • Case TS014232963
  • Case TS014348824

· One min read

Work in Progress​

  • While examining the logs, the team noticed an error 413 “payload too large” associated with NGINX.
  • So, during this session, the team added a NGINX annotation within the CR to fix this error.
  • After some testing, the team was successfully able to add content to a document and no longer encountered the web client error.

Currently Tracking​

  • The team started troubleshooting FileNet Navigator. Navigator wont launch after login. This seems to be tied to a ingress error.
  • The team is gathering logs from this session and will work closely with IBM support on this issue.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014370797

Cases closed: 2

  • Case TS014232963
  • Case TS014348824

· One min read

Work in Progress​

  • Today the team verified the java certs that were associated with Process Designer.
  • After the java certs were updated, the team was able to launch Process Designer.
  • The team then walked the client through WSDL setup and configuration in FileNet.

Currently Tracking​

  • While the team was able to get Process Designer working, the main error of FileNet being unable to connect to the web client persisted. The web client error appears to be a separate error and we are continuing to work with IBM support on this issue.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014370797

Cases closed: 2

  • Case TS014232963
  • Case TS014348824

· One min read

Work in Progress​

  • With the help of IBM support, the team checked the settings of the Advanced Storage Area and devices.
  • The team noticed FileNet had 3 virtual servers listed but there were only 2 replicas. The team tried scaling to 3 replicas, however the error persisted.
  • The team then gathered logs from process designer for further investigation.

Currently Tracking​

  • We will continue to work closely with IBM support and have provided them logs of today’s session.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014370797

Cases closed: 2

  • Case TS014232963
  • Case TS014348824

· One min read

Work in Progress​

  • With the help of IBM support, the team investigated network tracing using various browsers.
  • The team noticed that creating a document with content resulted in an error, while an empty document did not.
  • The team also verified the network policies within the cluster.
  • The team then collected more logs for further troubleshooting.

Currently Tracking​

  • We will continue to work closely with IBM support and have provided them logs of today’s session.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014370797

Cases closed: 2

  • Case TS014232963
  • Case TS014348824

· One min read

Work in Progress​

  • After upgrading the client's environment to version 5.5.11, the team verified the upgrade, and the pods are now in a healthy state.
  • With IBM support on today’s session, the team recreated the FileNet error that we were previously encountering.
  • The team gathered logs from the client's environment and will be working closely with support to evaluate the errors and logs.

Currently Tracking​

  • We will continue to work closely with IBM support and have provided them logs of today’s session.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014370797

Cases closed: 2

  • Case TS014232963
  • Case TS014348824

· One min read

Work in Progress​

  • Today the team continued to upgrade the client’s FileNet environment to version 5.5.11.
  • During the session, the team was able to successfully update the operator.
  • The team then updated the CR to reflect the new image tags and added the ‘sc_disable_read_only_root_filesystem’ variable.

Currently Tracking​

  • The client is watching to see if the rolling update of the running pods is successful.
  • We will continue to work closely with IBM support.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014370797

Cases closed: 2

  • Case TS014232963
  • Case TS014348824

· One min read

Work in Progress​

  • Today the team began to upgrade the client’s FileNet environment to version 5.5.11.
  • There were some errors while applying the update and the team spent time troubleshooting. -The team encountered a resource error that appears to be the same error the team faced during initial setup.

Currently Tracking​

  • The team will investigate how the resource error was previously resolved and will attempt to implement the solution in the next session.
  • We will continue to work closely with IBM support on these errors.

Next Steps​

Tracking​

Cases open: 1

  • Case TS014370797

Cases closed: 2

  • Case TS014232963
  • Case TS014348824