Agenda

GSoC 2023 season has just started and it is a good opportunity to contribute and learn about open source.
ASF has been participating in the event for several years now and will also be participating this year.

This session is targeted towards college students in Indore, to spread awareness about GSoC and guide them through the Apache Way and how the ASF works at large to facilitate participation in the ASF projects for this season.
We aspire to bring in more contributions to the Apache projects by developing interest and awareness in the college students.

This will be an in-person meetup scheduled on 10th February 2023, Friday.

We will be covering the following topics:

1. Introduction to GSoC
- Why they should participate
- GSoC 2023 timeline
- Tips on how to approach a mentoring organization
- Tips on how to sail through the selection process

(Speaker: Priya Sharma, PMC ComDev)

2. GSoC with the ASF
- Open Source
- What is ASF and its ties with GSoC
- The Apache Way
- How Apache projects community work
- How to set up and get started with a project

(Speaker: Aditya Sharma, ASF Member)

Date : 10 Feb 2023

Venue :

HotWax Systems,

80, Scheme Number 78, Part II,
Vijay Nagar,
Indore, Madhya Pradesh
452010
https://goo.gl/maps/3XKrbuXUd3pYcDFE7

Time : 10:30 AM IST to 12:30 PM IST

Gsoc-with-ASF.pdf

GSoC 2023 Idea List GSoC

Key Summary required-knowledge difficulty-level potential-mentor Description
STATISTICS-78 Create JMH benchmarks for alternative summation algorithms Medium

Develop JMH benchmarks to compare:

  1. Simple Sum (simply add the input values)
  2. Kahan summation algorithm (https://en.wikipedia.org/wiki/Kahan_summation_algorithm)
  3. Two sum method (Neumaier sum) https://en.wikipedia.org/wiki/Kahan_summation_algorithm#Further_enhancements
  4. Higher-order extended precision floating-point sums
  5. The add() methods in `BigDecimal` class with various MathContext limits on the precision.
STATISTICS-77 Implement Sum Medium

This implementation uses commons-numbers Sum as the underlying implementation.

This may be further expanded to include various other Sum implementations like Kahan Sum, Neumaier Sum (Two sum method) etc. with the option for the users to choose an implementation. However, before beginning the implementation of various other algorithms it may be worthwhile to discuss and investigate the relative speed of the current Sum implementation compared to DoubleStream.sum through JMH benchmarks.
When we know the different speeds, it would be more clear if the user should have a choice for this statistic, e.g.

  • When using DoubleStream.sum is too slow and it would be nice to have a simple sum variation.
  • When the sum has a lot of cancellation and DoubleStream.sum suffers from random walk around zero.
STATISTICS-76 Implement Max Easy Logic to compute the maximum of all the input values.
STATISTICS-75 Create JMH benchmarks for statistic initialization Easy Create JMH benchmarks for statistic initialization comparing Arrays.stream to a simple for loop.
STATISTICS-73 Create a new o.a.c.s.descriptive module Easy Create a new commons-statistics-descriptive module which contains the implementation of all univariate statistics.
STATISTICS-72 Implement Min Easy Logic to compute the minimum of all values seen so far.
STATISTICS-71 Implementation of Univariate Statistics Medium Alex Herbert, Gilles Sadowski

Jira ticket to track the implementation of the Univariate statistics required for the updated SummaryStatistics API.
The implementation would be "storeless". It should be used for calculating statistics that can be computed in one pass through the data without storing the sample values.

Currently I have the definition of API as (this might evolve as I continue working)

public interface DoubleStorelessUnivariateStatistic extends DoubleSupplier {

DoubleStorelessUnivariateStatistic add(double v);

long getCount();

void combine(DoubleStorelessUnivariateStatistic other);
}
STATISTICS-54 [GSoC] Summary statistics API for Java 8 streams

Placeholder for tasks that could be undertaken in this year's GSoC.

Ideas:

  • Design an updated summary statistics API for use with Java 8 streams based on the summary statistic implementations in the Commons Math stat.descriptive package including moments, rank and summary sub-packages.
NUMBERS-193 Add support for extended precision floating-point numbers Easy

Add implementations of extended precision floating point numbers.

An extended precision floating point number is a series of floating-point numbers that are non-overlapping such that:

``` double-double (a, b): |a| > |b| a == a + b ```

Common representations are double-double and quad-double (see for example David Bailey's paper on a quad-double library: QD).

Many computations in the Commons Numbers and Statistics libraries use extended precision computations where the accumulated error of a double would lead to complete cancellation of all significant bits; or create intermediate overflow of integer values.

This project would formalise the code underlying these use cases with a generic library applicable for use in the case where the result is expected to be a finite value and using Java's BigDecimal and/or BigInteger negatively impacts performance.

An example would be the average of long values where the intermediate sum overflows or the conversion to a double loses bits:

``` long[] values = {Long.MAX_VALUE, Long.MAX_VALUE}; System.out.println(Arrays.stream(values).average().getAsDouble()); System.out.println(Arrays.stream(values).mapToObj(BigDecimal::valueOf) .reduce(BigDecimal.ZERO, BigDecimal::add) .divide(BigDecimal.valueOf(values.length)).doubleValue()); long[] values2 = {Long.MAX_VALUE, Long.MIN_VALUE}; System.out.println(Arrays.stream(values2).asDoubleStream().average().getAsDouble()); System.out.println(Arrays.stream(values2).mapToObj(BigDecimal::valueOf)    .reduce(BigDecimal.ZERO, BigDecimal::add) .divide(BigDecimal.valueOf(values2.length)).doubleValue()); ```

Outputs:

-1.0
9.223372036854776E18
0.0
-0.5
NEMO-485 Efficient Dynamic Reconfiguration in Stream Processing In Stream processing, we have many methods, starting from the primitive checkpoint-and-replay to a more fancy version of reconfiguration and reinitiation of stream workloads. We aim to find a way to find the most effective and efficient way of reconfiguring stream workloads. Sub-issues are to be created later on.
NEMO-479 Approximate the factors that affect the stage group level execution time There are some factors that can affect the stage group level simulation, such as a latency, the rate of skewed data and the error rate of the executor etc. It is required to find a reasonable distribution form for these factors. Such as the normal distribution or the landau distribution. In actual running, It makes it possible to approximate the model with a small amount of data.
NEMO-478 Implement an Accurate Simulator based on Functional model

Missing a deadline often has significant consequences for the business. And simulator can contributes to other approach for optimization

So Implement a Simulator for Stream Processing Based on Functional models.

There are some requirements:

  • Simulation should be able to execute before or during job execution
  • When a simulation is executed during job is running, It must be fast enough not to affect the job.
  • Information about running environment is received through argument.
  • At least network topology should be considered for the WAN environment.
NEMO-477 Implement a model that represent a task level exeuction time with statistical analysis

The current SimulatedTaskExecutor is hardly available. because it needs actual metric to predict execution time. To increase utilization, we need new model that predicts a task level execution time with statistical analysis.

Some of the related TODOs are as follows:

  • Find factors that affect a task level execution time. with loose grid search.
  • Infer the most suitable model with tight grid search.
NEMO-471 Evaluate the performance of Work Stealing implementation
NEMO-468 Detect skewed task periodically
NEMO-467 Collect task statistics necessary for estimating duration
NEMO-466 Dynamic Work Stealing on Nemo for handling skews

We aim to handle the problem on throttled resources (heterogeneous resources) and skewed input data. In order to solve this problem, we suggest dynamic work stealing that can dynamically track task statuses and steal workloads among each other. To do this, we have the following action items:

  • Dynamically collecting task statistics during execution
  • Detecting skewed tasks periodically
  • Splitting the data allocated in skewed tasks and reallocating them into new tasks
  • Synchronizing the optimization procedure
  • Evaluation of the resulting implementations
NEMO-449 Runtime Level Caching Mechanism

If the the compile time identifies what data can be cached, the runtime requires logic to make this happen.

Implementation needs:

  • (Driver) receive and update the status of blocks from various Executors, right now this seems to be best implemented as part of BlockManagerMaster
  • (Driver) communicate to the Executors the availability, location and status of blocks
  • Possible concurrency issues:
  1. Concurrency in Driver when multiple Executors update/inquire the same block information
  2. Concurrency in Executor when a single cached block is accessed simultaneously.
NEMO-444 Implement spill mechanism on Nemo Currently, Nemo doesn't have a spill mechanism. This makes executors prone to memory problems such as OOM(Out Of Memory) or GC when task data is large. For example, handling skewed shuffle data in Nemo results in OOM and executor failure, as all data has to be handled in-memory.
We need to spill in-memory data to secondary storage when there are not enough memory in executor.
NEMO-443 Efficient Caching and Spilling on Nemo

In-memory caching and spilling are essential features in in-memory big data processing frameworks, and Nemo needs one.

  • Identify and persist frequently used data and unpersist it when its usage ended
  • Spill in-memory data to disk upon memory pressure
NEMO-441 Dynamic Task Sizing on Nemo This is an umbrella issue to keep track of the issues related to the dynamic task sizing feature on Nemo.
Dynamic task sizing needs to consider a workload and try to decide on the optimal task size based on the runtime metrics and characteristics. It should have an effect on the parallelism and the partitions, on how many partitions an intermediate data should be divided/shuffled into, and to effectively handle skews in the meanwhile.
NEMO-425 Nemo on Google Dataproc Issues for making it easy to install and use Nemo on Google Dataproc.
NEMO-389 Enhance Nemo to support autoscaling for bursty loads The load of streaming jobs usually fluctuate according to the input rate or operations (e.g., window). Supporting the automatic scaling could reduce the operational cost of running streaming applications, while minimizing the performance degradation that can be caused by the bursty loads.
We can harness the cloud resources such as VMs and serverless frameworks to acquire computing resources on demand. To realize the automatic scaling, the following features should be implemented.
1) state migration: scaling jobs require moving tasks (or partitioning a task to multiple ones). In this situation, the internal state of the task should be serialized/deserialized.
2) input/output rerouting: if a task is moved to a new worker, the input and output of the task should be redirected.
3) dynamic Executor or Task creation/deletion: Executor}}s or {{Task can be dynamically created or deleted.
4) scaling policy: a scaling policy that decides when and how to scale out/in should be implemented.
MATH-1655 Refactoring of GA functionality Medium

As discussed extensively on the "dev" ML[1][2], there are two competing designs (please review them on the dedicated git branch) for the refactoring of the basic functionality currently implemented in the org.apache.commons.math4.legacy.genetics "legacy" package.

TL;DR;

  • The discussion has pointed to major (from a maintenance POV) issues of the design proposed by the OP.
  • The alternative (much simpler) design has been implemented as proof-of-concept (indicating that some corner might have been cut).
  • The OP mentioned correctness issues in the "simple" design but did neither fix them nor provided answers on the LM to that effect.
  • Questions concerning other possible "bloat" (e.g. on using a custom representation of the "binary chromosome" concept instead of the BitSet available from the JDK) were also left dangling.
  • Refactoring of the "basic" GA functionality (the purpose of the "proof-of-concept") must be decoupled from the new feature which the OP wanted to implement ("adaptive probability generation").
  • Unit tests (a.o. all those from the "legacy" code) must demonstrate that the refactored code does (or does not) behave correctly, and bugs should be fixed on the "simple" implementation, before implementing the new feature on top of it.

[1] https://markmail.org/message/qn7gq2y7xjoxukzp
[2] https://markmail.org/message/f66iii3a4kmjaprr

MATH-1641 [GSoC] Update components including machine learning; linear algebra; special functions Alex Herbert, Gilles Sadowski

Placeholder for tasks that could be undertaken in this year's GSoC.

Ideas (extracted from the "dev" ML):

  1. Redesign and modularize the "ml" package
    -> main goal: enable multi-thread usage.
  2. Abstract the linear algebra utilities
    -> main goal: allow switching to alternative implementations.
  3. Redesign and modularize the "random" package
    -> main goal: general support of low-discrepancy sequences.
  4. Refactor and modularize the "special" package
    -> main goals: ensure accuracy and performance and better API,
    add other functions.
  5. Upgrade the test suite to Junit 5
    -> additional goal: collect a list of "odd" expectations.
  6. Review and finalize pending issues about the refactoring of the "genetic algorithm" functionality (cf. dedicated branch)

Other suggestions welcome, as well as

  • delineating additional and/or intermediate goals,
  • signalling potential pitfalls and/or alternative approaches to the intended goal(s).
JAMES-3819 [GSOC] James as a (distributed) MX server

Why ?

Alternatives like Postfix...

  • Do not offer a unified view of the mail queue across nodes
  • Requires statefull persistent storage

Given Apache James recent push to adopt a distributed mail queue based on Pulsar supporting delays (JAMES-3687), it starts making sense developing tooling for MX related tooling.

I propose myself to mentor a Gsoc on this topic.

Benefits for the student

At the end of this GSOC you will...

  • Have a solid understanding of email relaying and associated mechanics
  • Understand James modular architecture (mailet/ matcher / routes)
  • Have a hands-on expertise in SQL / NoSQL working with technologies like Cassandra, Redis, JPA...
  • Identify fix and solve architecture problems.
  • Conduct performance tests and develop an operational mindset

Inventory...

James ships a couple of MX related tools within smtp-hooks/mailets in default packages. It would make sense to me to move those as an extension.

James supports today...

checks agains DNS blacklists. `DNSRBLHandler` or `URIRBLHandler` smtp hook for instance. This can be moved as an extension IMO.

We would need a little performance benchmark to document performance implications of activating DNS-RBL.

Finally as quoted by a gitter guy: it would make more sens to have this done as a MailHook rather as a RcptHook as it would avoid doing the same job again and over again for each recipients. See JAMES-3820 .

Grey listing. There's an existing implementation using JDBC as an underlying storage.

Move it as an extension.

Remove JDBC storage, propose 2 storage possibilities: in-memory for single node, REDIS for a distributed topology.

Some work around whitelist mailets? Move it as an extension, propose JPA, Cassandra, and XML configured implementations ? With a route to manage entries in there for JPA + Cassandra ?

I would expect a student to do his own little audit and come up with extra suggestions!

JAMES-3695 Adopt Pulsar as the messaging technology backing the distributed James server

https://www.mail-archive.com/server-dev@james.apache.org/msg71462.html

A good long term objective for the PMC is to drop RabbitMQ in
favor of pulsar (third parties could package their own components using
RabbitMQ if they wishes...)

This means:

  • Solve the bugs that were found during the Pulsar MailQueue review
  • Pulsar MailQueue need to allow listing blobs in order to be
    deduplication friendly.
  • Provide an event bus based on Pulsar
  • Provide a task manager based on Pulsar
  • Package a distributed server backed by pulsar, deprecate then replace
    the current one.
  • (optionally) support mail queue priorities

While contributions would of course be welcomed on this topic, we could
offer it as part of GSOC 2022, and we could co-mentor it with mentors of
the Pulsar community (see [3])

[3] https://lists.apache.org/thread/y9s7f6hmh51ky30l20yx0dlz458gw259

Would such a plan gain traction around here ?

JAMES-1931 Implement a web ui for James administration James today provides a command line tool to do administration tasks like creating a domain, listing users, setting quota, etc.
It requires access to JMX port and even if lot of admins are confortable with such tools, to make our user base broader, we probably should expose the same commands in Rest and provide a fancy default web ui.
The task would need some basic skills on frontend tools to design an administration board, knowledge on what REST mean and enough Java understanding to add commands to existing Rest backend.
In the team, we have a strong focus on test (who want a mail server that is not tested enough ?) so we will explain and/or teach the student how to have the right test coverage of the features using modern tools like Cucumber, Selenium, rest-assured, etc.
IMAGING-346 Placeholder for 1.0 release A placeholder ticket, to link other issues and organize tasks related to the 1.0 release of Commons Imaging.
The 1.0 release of Commons Imaging has been postponed several times. Now we have a more clear idea of what's necessary for the 1.0 (see issues with fixVersion 1.0 and 1.0-alpha3, and other open issues), and the tasks are interesting as it involves both basic and advanced programming for tasks such as organize how test images are loaded, or work on performance improvements at byte level and following image format specifications.
The tasks are not too hard to follow, as normally there are example images that need to work with Imaging, as well as other libraries in C, C++, Rust, PHP, etc., that process these images correctly. Our goal with this issue is to a) improve our docs, b) improve our tests, c) fix possible security issues, d) get the parsers in Commons Imaging ready for the 1.0 release.
Assigning the label for GSoC 2023, and full time. Although it would be possible to work on a smaller set of tasks for 1.0 as a part time too.
GSOC-250 Apache SeaTunnel(Incubating): Http Client For SeaTunnel Zeta

Apache SeaTunnel(Incubating)

SeaTunnel is a very easy-to-use ultra-high-performance distributed data integration platform that supports real-time synchronization of massive data. It can synchronize tens of billions of data stably and efficiently every day, and has been used in the production of nearly 100 companies.

SeaTunnel provides a Connector API that does not depend on a specific execution engine. Connectors (Source, Transform, Sink) developed based on this API can run On many different engines, such as SeaTunnel Zeta, Flink, Spark that are currently supported. SeaTunnel has supported more than 100 Connectors, and the number is surging.

Website: https://seatunnel.apache.org/

GitHub: https://github.com/apache/incubator-seatunnel

Background

To use SeaTunnel, the current user needs to first create and write a config file that specifies the engine that runs the job, as well as engine related parameters. Then define the Source, Transform, and Sink of the job. We hope to provide a client that allows users to define the engine, Source, Transform, and Sink information of the job run directly through code in the client without having to start with a config file. The user can then submit the job definition information through the Client, and SeaTunnel will run these jobs. After the job is submitted, the user can obtain the status of the job running through the Client. For jobs that are already running, users can use this client to manage them, such as stopping jobs, temporary jobs, and so on.

Task

1. Discuss with the mentors what you need to do

2. Learn the details of the Apache SeaTunnel project

3. Discuss and complete design and development

Relevant Skills

  1. Familiar with Java, Http
  2. Familiar with SeaTunnel is better

Mentor

  • Mentor: Jun Gao, Apache SeaTunnel(Incubating) PPMC Member, gaojun2048@apache.org
  • Mentor: Li Liu, Apache SeaTunnel(Incubating) Commiter, ic4y@apache.org
  • Mailing List: dev@seatunnel.apache.org
GSOC-249 [SkyWalking] Build the OAP into GraalVM native image Currently skywalking OAP is bundled as a tar ball when releasing, and the start time is long, we are looking for a way to distribute the binary executable in a more convenient way and speed up the bootstrap time. Now we found that GraalVM is a good fit not only it can solve the two aforementioned points but also it will bring benefits that, we can rewrite our LAL or even MAL system in the future with a more secure and isolated method, wasm, which is supported GraalVM too!
so this task is to adjust OAP, build it into GraalVM and make all tests in OAP passed.
GSOC-248 Apache ShenYu Gsoc 2023 - ShenYu WasmPlugin

Background:{}

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance. Currently, ShenYu has good scalability in the Java language. However, ShenYu's support for multiple languages is still relatively weak.

The wasm bytecode is designed to be encoded in a size- and load-time-efficient binary format. WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities available on a wide range of platforms.

The goal of WasmPlugin is to be able to run wasm bytecode(wasmer-java is a good choice, if you find a better choice, please discuss with me), and other languages can write ShenYu plugins based on this language(such as Rust/golang/C++) as long as they can be compiled into wasm bytecode.

More documents on wasm and WASI are as follows:
https://github.com/WebAssembly/design
https://github.com/WebAssembly/WASI

Relevant Skills

Know the use of Apache ShenYu, especially the plugin
Familiar with Java and another language which can be compiled into wasm bytecode

Task List

1.develop shenyu-wasm-plugin.

2.write integrated test for shenyu-wasm-plugin.

3.write wasm plugin documentation in shenyu-website.

Links:

website: https://shenyu.apache.org/

issues: https://github.com/apache/shenyu/issues/4492

GSOC-247 Apache ShardingSphere: Introduce JVM chaos to ShardingSphere

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere

Background

There is a proposal about the background of ChaosEngineering as belows:

Introduce ChaosEngineering for ShardingSphere #32
And we also proposed a generic controller for ShardingSphereChaos as belows:

[GSoC 2023] Introduce New CRD ShardingSphereChaos #272
The ShardingSphereChaos controller is aiming at different chaos tests. This JVMChaos is an important one.

Task

Write several scripts to implement different JVMChaos for main features of ShardingSphere. The specific case list is as follows.

  • Add scripts injecting chaos to DataSharding
  • Add scripts injecting chaos to ReadWritingSplitting
  • Add scripts injecting chaos to DatabaseDiscovery
  • Add scripts injecting chaos to Encryption
  • Add scripts injecting chaos to Mask
  • Add scripts injecting chaos to Shadow
    Basically, these scripts will cause unexpected behaviour while executing the related. DistSQL.

Relevant Skills

  • Master Go language, Ginkgo test framework
  • Have a deep understanding of Apache ShardingSphere concepts and practices.
  • JVM byte mechanisms like ByteMan, ByteBuddy.

Targets files

JVMChaos Scripts - https://github.com/apache/shardingsphere-on-cloud/chaos/jvmchaos/scripts/

Mentor
Liyao Miao, Committer of Apache ShardingSphere, miaoliyao@apache.org
Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org

GSOC-246 Apache ShardingSphere: Write a converter to generate DistSQL

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere

Background

Currently we try to promote StorageNode as major CRD to represent a set of storage units for ShardingSphere.

Task

The elementary task is that the storage node controller could manage the lifecycle of a set of storage units, like PostgreSQL, in kubernetes.

We don't hope to create another wheel like pg-operator. So consider using a predefined parameter group to generate the target CRD.

  • [ ] Generate DistSQL according to the Golang struct `EncryptionRule`
  • [ ] Generate DistSQL according to the Golang struct `ShardingRule`
  • [ ] Generate DistSQL according to the Golang struct `ReadWriteSplittingRule`
  • [ ] Generate DistSQL according to the Golang struct `MaskRule`
  • [ ] Generate DistSQL according to the Golang struct `ShadowRule`

    Relevant Skills

1. Master Go language, Ginkgo test framework
2. Have a basic understanding of Apache ShardingSphere Concepts and DistSQL

Targets files

DistSQL Converter - https://github.com/apache/shardingsphere-on-cloud/blob/main/shardingsphere-operator/pkg/distsql/converter.go, etc.

Example

A struct defined as below:

```golang
type EncryptRule struct{}
func (t EncryptRule) ToDistSQL() string {}
```
While invoking ToDistSQL() it will generate a DistSQL regarding a EncryptRule like:

```SQL
CREATE ENCRYPT RULE t_encrypt (....
```

References:

Mentor
Liyao Miao, Committer of Apache ShardingSphere, miaoliyao@apache.org

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org

GSOC-245 Apache ShardingSphere: Introduce new CRD as StorageNode for better usability

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere

Background

There is a proposal about new CRD Cluster and ComputeNode as belows:

  • #167
  • #166

Currently we try to promote StorageNode as major CRD to represent a set of storage units for ShardingSphere.

Task

The elementary task is that the storage node controller could manage the lifecycle of a set of storage units, like PostgreSQL, in kubernetes.

We don't hope to create another wheel like pg-operator. So consider using a predefined parameter group to generate the target CRD.

  • [ ] Create a PostgreSQL cluster while a StorageNode with pg parameters is created
  • [ ] Update the PostgreSQL cluster while updated StorageNode
  • [ ] Delete the PostgreSQL cluster while deleted StorageNode. Notice this may need a deletion strategy.
  • [ ] Reconciling StorageNode according to the status of PostgreSQL cluster.
  • [ ] The status of StorageNode would be consumed by common storage units related DistSQLs

Relevant Skills

1. Master Go language, Ginkgo test framework
2. Have a basic understanding of Apache ShardingSphere Concepts
3. Be familiar with Kubernetes Operator, kubebuilder framework

Targets files

StorageNode Controller - https://github.com/apache/shardingsphere-on-cloud/blob/main/shardingsphere-operator/pkg/controllers/storagenode_controller.go

Mentor

Liyao Miao, Committer of Apache ShardingSphere, miaoliyao@apache.org

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org

GSOC-244 Apache ShardingSphere: Introduce New CRD ShardingSphereChaos

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere

Background

There is a proposal about the background of ChaosEngineering as belows:

The ShardingSphereChaos controller is aiming at different chaos tests.

Task

Propose a generic controller for ShardingSphereChaos, which reconcile CRD ShardingSphereChaos, prepare, execute and verify test.

  • [ ] Support common ShardingSphere features, prepare test rules and dataset
  • [ ] Generating chaos type according to the backend implementation
  • [ ] Verify testing result with DistSQL or other tools

Relevant Skills

1. Master Go language, Ginkgo test framework
2. Have a deep understanding of Apache ShardingSphere concepts and practices.
3. Kubernetes operator pattern, kube-builder

Targets files

ShardingSphereChaos Controller - https://github.com/apache/shardingsphere-on-cloud/shardingsphere-operator/pkg/controllers/chaos_controller.go, etc.

Mentor

Liyao Miao, Committer of Apache ShardingSphere, miaoliyao@apache.org

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org

GSOC-243 [GSoC][Teaclave (incubating)] Data Privacy Policy Definition and Function Verification

Background

The Apache Teaclave (incubating) is a cutting-edge solution for confidential computing, providing Function-as-a-Service (FaaS) capabilities that enable the decoupling of data and function providers. Despite its impressive functionality and security features, Teaclave currently lacks a mechanism for data providers to enforce policies on the data they upload. For example, data providers may wish to restrict access to certain columns of data for third-party function providers. Open Policy Agent (OPA) offers flexible control over service behavior and has been widely adopted by the cloud-native community. If Teaclave were to integrate OPA, data providers could apply policies to their data, enhancing Teaclave’s functionality. Another potential security loophole in Teaclave is the absence of a means to verify the expected behavior of a function. This gap leaves the system vulnerable to exploitation by malicious actors. Fortunately, most of Teaclave’s interfaces can be reused, with the exception of the function uploading phase, which may require an overhaul to address this issue. Overall, the integration of OPA and the addition of a function verification mechanism would make Teaclave an even more robust and secure solution for confidential computing.

Benefits

If this proposal moves on smoothly, new functionality will be added to the Teaclave project that enables the verification of the function behavior that it strictly conforms to a prescribed policy.

Deliverables

  • Milestones: Basic policies (e.g., addition, subtraction) of the data can be verified by Teaclave; Complex policies can be verified.
  • Components: Verifier for the function code; Policy language adapters (adapt policy language to verifier); Policy language parser; Function source code converter (append policies to the functions).
  • Documentation: The internal working mechanism of the verification; How to write policies for the data.

Timeline Estimation

  • 0.5 month: Policy language parser and/or policy language design (if Rego is not an ideal choice).
  • 1.5 − 2 months: Verification contracts rewriting on the function source code based on the policy parsed. • (∼ 1 month): The function can be properly verified formally (by, e.g., querying the Z3 SMT solver).

Difficulty: Major
Project size: ~350 hour (large)
Potential mentors:
Mingshen Sun, Apache Teaclave (incubating) PPMC, mssun@apache.org

GSOC-242 [GSoC][Airflow] Automation for PMC This is a project to implement a tool for PMC task automation.
This is a large project.
Mentor will be aizhamal ,
GSOC-241 [GSoC][Beam] Advancing the Rust SDK on Beam

Beam has an experimental, ongoing implementation for a Rust SDK.

This project involves advancing that implementation and making sure it's compiant with Beam standards.

Good resource materials:

This project is large.

GSOC-240 [GSoC][Beam] Advancing the Beam-on-Ray runner

There is a community effort to build a Beam runner to run Beam pipelines on top of Ray: https://github.com/ray-project/ray_beam_runner/

This involves pushing that project forward. It will require writing lots of Python code, and specifically going through the list of issues (https://github.com/ray-project/ray_beam_runner/issues) and solving as many of them as possible to make sure the runner is compliant.

Good resource docs:

This project is large.

GSOC-169 Apache ShardingSphere: Add the feature of switching logging framework

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere

Background

ShardingSphere provides two adapters: ShardingSphere-JDBC and ShardingSphere-Proxy.

Now, ShardingSphere uses logback for logging, but consider the following situations:

  • Users may need to switch the logging framework to meet special needs, such as log4j2 can provide better asynchronous performance;
  • When using the JDBC adapter, the user application may not use logback, which may cause some conflicts.

Why doesn't the log facade suffice? Because ShardingSphere provides users with clustered logging configurations (such as changing the log level online), this requires dynamic construction of logger, which cannot be achieved with only the log facade.

Task

1. Design and implement logging SPI to support multiple logging frameworks (such as logback and log4j2)
2. Allow users to choose which logging framework to use through the logging rule

Relevant Skills

1. Master JAVA language

2. Basic knowledge of logback and log4j2

3. Maven

Mentor

Longtao Jiang, Committer of Apache ShardingSphere, jianglongtao@apache.org

Trista Pan, PMC of Apache ShardingSphere, panjuan@apache.org

GSOC-168 Apache ShenYu Gsoc 2023 - ShenYu End-To-End SpringCloud plugin test case

Background:

Shenyu is a native API gateway for service proxy, protocol translation and API governance. but Shenyu lack of End-To-End Tests.

Relevant skills:

1.Understand the architecture of ShenYu

2.Understand SpringCloud micro-service and ShenYu SpringCloud proxy plugin.

3.Understand ShenYu e2e framework and architecture.

How to coding

1.please refer to org.apache.shenyu.e2e.testcase.plugin.DividePluginCases

How to test

1.start shenyu admin in docker

2.start shenyu boostrap in docker

3.run test case org.apache.shenyu.e2e.testcase.plugin.PluginsTest#testDivide

Task List

1.develop e2e tests of the springcloud plug-in.

2.write shenyu e2e springcloud plugin documentation in shenyu-website.

3.refactor the existing plugin test cases.

Links:

website: https://shenyu.apache.org/

issues: https://github.com/apache/shenyu/issues/4474

GSOC-165 Apache ShardingSphere: Support mainstream database metadata table query

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere

Background

ShardingSphere has designed its own metadata database to simulate metadata queries that support various databases.

More details:

https://github.com/apache/shardingsphere/issues/21268
https://github.com/apache/shardingsphere/issues/22052

Task

  • Support PostgreSQL And openGauss `\d tableName`
  • Support PostgreSQL And openGauss `\d+`
  • Support PostgreSQL And openGauss `\d+ tableName`
  • Support PostgreSQL And openGauss `l`
  • Support query for MySQL metadata `TABLES`
  • Support query for MySQL metadata `COLUMNS`
  • Support query for MySQL metadata `schemata`
  • Support query for MySQL metadata `ENGINES`
  • Support query for MySQL metadata `FILES`
  • Support query for MySQL metadata `VIEWS`

Notice, these issues can be a good example.

https://github.com/apache/shardingsphere/pull/22053
https://github.com/apache/shardingsphere/pull/22057/
https://github.com/apache/shardingsphere/pull/22166/
https://github.com/apache/shardingsphere/pull/22182

Relevant Skills

  • Master JAVA language
  • Have a basic understanding of Zookeeper
  • Be familiar with MySQL/Postgres SQLs

Mentor

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org

Zhengqiang Duan, PMC of Apache ShardingSphere, duanzhengqiang@apache.org

GSOC-161 Code Insights for Apache StreamPipes

Apache StreamPipes

Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.

Background

StreamPipes has grown significantly throughout recent years. We were able to introduce a lot of new features and attracted both users and contributors. Putting the cherry on the cake, we were graduated as an Apache top level project in December 2022. We will of course continue developing new features and never rest to make StreamPipes even more amazing. Although, since we are approaching with full stream towards our `1.0` release, we want to project also to get more mature. Therefore, we want to address one of our Achilles' heels: our test coverage.

Don't worry, this issue is not about implementing myriads of tests for our code base. As a first step, we would like to make the status quo transparent. That means we want to measure our code coverage consistently across the whole codebase (Backend, UI, Python library) and report the coverage to codecov. Furthermore, to benchmark ourselves and motivate us to provide tests with every contributing, we would like to lock the current test coverage as an lower threshold that we always want to achieve (meaning in case we drop CI builds fail etc). With time we then can increase the required coverage lever step to step.

More than monitoring our test coverage, we also want to invest in better and more clean code. Therefore, we would like to adopt sonarcloud for our repository.

Tasks

  • [ ] calculate test coverage for all main parts of the repo
  • [ ] send coverage to codeCov
  • [ ] determine coverage threshold and let CI fail if below
  • [ ] include sonarcloud in CI setup
  • [ ] include automatic coverage report in PR validation (see an example here ) -> optional
  • [ ] include automatic sonarcloud report in PR validation -> optional
  • [ ] what ever comes in your mind 💡 further ideas are always welcome

❗Important Note❗

Do not create any account in behalf of Apache StreamPipes in Sonarcloud or in CodeCov or using the name of Apache StreamPipes for any account creation. Your mentor will take care of it.

Relevant Skills

  • basic knowledge about GitHub worfklows

Learning Material

References

You can find our corresponding issue on GitHub here

Name and Contact Information

Name: Tim Bossenmaier

email: bossenti[at]apache.org

community: dev[at]streampipes.apache.org

website: https://streampipes.apache.org/

GSOC-158 Apache EventMesh: Optimize the event-bridge on EventMesh Apache EventMesh (incubating)
Apache EventMesh is a fully serverless platform used to build distributed event-driven applications.
Website: https://eventmesh.apache.org
GitHub: https://github.com/apache/incubator-eventmesh
Upstream Issue: https://github.com/apache/incubator-eventmesh/issues/3494
Background
Through eventmesh‘s event bridge feature, we can connect data to heterogeneous data storage, we hope that the community can optimize the current eventbridge capability of EventMesh to realize the data connection of different event stores.
Task
1. Discuss with the mentors what you need to do
2. Learn the details of the Apache EventMesh project
3. Verify the ability of different EventMesh cluster instances to synchronize data, sort out the corresponding verification step documents, and optimize the current EventMesh bridge features
Recommended Skills
1. Familiar with Java
2. Familiar with MQ is better
Mentor
Eason Chen, PPMC of Apache EventMesh, https://github.com/qqeasonchen, chenguangsheng@apache.org
Mike Xue, PPMC of Apache EventMesh, https://github.com/xwm1992, mikexue@apache.org
GSOC-157 Apache ShardingSphere: Enhance ComputeNode reconciliation

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere

Background

There is a proposal about new CRD Cluster and ComputeNode as belows:

Currently we try to promote ComputeNode as major CRD to represent a special ShardingSphere Proxy deployment. And plan to use Cluster indicating a special ShardingSphere Proxy cluster.

Task

This issue is to enhance ComputeNode reconciliation availability. The specific case list is as follows.

  • Add IT test case for Deployment spec volume
  • Add IT test case for Deployment spec template init containers
  • Add IT test case for Deployment spec template spec containers
  • Add IT test case for Deployment spec volume mounts
  • Add IT test case for Deployment spec container ports
  • Add IT test case for Deployment spec container image tag
  • Add IT test case for Service spec ports
  • Add IT test case for ConfigMap data serverconfig
  • Add IT test case for ConfigMap data logback

    Notice, these issues can be a good example.
  • chore: add more Ginkgo tests for ComputeNode #203

Relevant Skills

  1. Master Go language, Ginkgo test framework
  2. Have a basic understanding of Apache ShardingSphere Concepts
  3. Be familiar with Kubernetes Operator, kubebuilder framework

Targets files

ComputeNode IT - https://github.com/apache/shardingsphere-on-cloud/blob/main/shardingsphere-operator/pkg/reconcile/computenode/compute_node_test.go

Mentor

Liyao Miao, Committer of Apache ShardingSphere, miaoliyao@apache.org

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org

GSOC-153 Apache ShardingSphere: Enhance SQLNodeConverterEngine to support more MySQL SQL statements

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere

Background

The ShardingSphere SQL federation engine provides support for complex SQL statements, and it can well support cross-database join queries, subqueries, aggregation queries and other statements. An important part of SQL federation engine is to convert the SQL statement parsed by ShardingSphere into SqlNode, so that Calcite can be used to implement SQL optimization and federated query.

Task

This issue is to solve the MySQL exception that occurs during SQLNodeConverterEngine conversion. The specific case list is as follows.

  • select_char
  • select_extract
  • select_from_dual
  • select_from_with_table
  • select_group_by_with_having_and_window
  • select_not_between_with_single_table
  • select_not_in_with_single_table
  • select_substring
  • select_trim
  • select_weight_string
  • select_where_with_bit_expr_with_ampersand
  • select_where_with_bit_expr_with_caret
  • select_where_with_bit_expr_with_div
  • select_where_with_bit_expr_with_minus_interval
  • select_where_with_bit_expr_with_mod
  • select_where_with_bit_expr_with_mod_sign
  • select_where_with_bit_expr_with_plus_interval
  • select_where_with_bit_expr_with_signed_left_shift
  • select_where_with_bit_expr_with_signed_right_shift
  • select_where_with_bit_expr_with_vertical_bar
  • select_where_with_boolean_primary_with_comparison_subquery
  • select_where_with_boolean_primary_with_is
  • select_where_with_boolean_primary_with_is_not
  • select_where_with_boolean_primary_with_null_safe
  • select_where_with_expr_with_and_sign
  • select_where_with_expr_with_is
  • select_where_with_expr_with_is_not
  • select_where_with_expr_with_not
  • select_where_with_expr_with_not_sign
  • select_where_with_expr_with_or_sign
  • select_where_with_expr_with_xor
  • select_where_with_predicate_with_in_subquery
  • select_where_with_predicate_with_regexp
  • select_where_with_predicate_with_sounds_like
  • select_where_with_simple_expr_with_collate
  • select_where_with_simple_expr_with_match

You need to compare the difference between actual and expected, and then correct the logic in SQLNodeConverterEngine so that actual can be consistent with expected.

After you make changes, remember to add case to SUPPORTED_SQL_CASE_IDS to ensure it can be tested.

Notice, these issues can be a good example.
https://github.com/apache/shardingsphere/pull/14492

Relevant Skills

1. Master JAVA language

2. Have a basic understanding of Antlr g4 file

3. Be familiar with MySQL and Calcite SqlNode

Targets files

SQLNodeConverterEngineIT

https://github.com/apache/shardingsphere/blob/master/test/it/optimizer/src/test/java/org/apache/shardingsphere/test/it/optimize/SQLNodeConverterEngineIT.java

Mentor

Zhengqiang Duan, PMC of Apache ShardingSphere, duanzhengqiang@apache.org

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org

Trista Pan, PMC of Apache ShardingSphere, panjuan@apache.org

GSOC-151 Apache EventMesh: Optimize the eventmesh-admin of EventMesh Apache EventMesh (incubating)
Apache EventMesh is a fully serverless platform used to build distributed event-driven applications.
Website: https://eventmesh.apache.org
GitHub: https://github.com/apache/incubator-eventmesh
Upstream Issue: https://github.com/apache/incubator-eventmesh/issues/3495
Background
At present, eventmesh-admin provides a management interface for eventmesh storage, but it only implements the management function of rocketmq, which needs to be further expanded. At the same time, it can provide CLI for users to quickly start and experience.
Task
1. Discuss with the mentors what you need to do
2. Learn the details of the Apache EventMesh project
3. Implement the management interface for other eventmesh storage
4. Support the CLI for quick start the eventmesh
Recommended Skills
1. Familiar with Java
2. Familiar with MQ is better
Mentor
Eason Chen, PPMC of Apache EventMesh, https://github.com/qqeasonchen, chenguangsheng@apache.org
Mike Xue, PPMC of Apache EventMesh, https://github.com/xwm1992, mikexue@apache.org
GSOC-150 OPC-UA browser for Apache StreamPipes

Apache StreamPipes

Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.

Background

StreamPipes is grown significantly throughout recent years. We were able to introduce a lot of new features and attracted both users and contributors. Putting the cherry on the cake, we were graduated as an Apache top level project in December 2022. We will of course continue developing new features and never rest to make StreamPipes even more amazing.

StreamPipes really shines when connecting Industrial IoT data. Such data sources typically originate from machine controllers, called PLCs (e.g., Siemens S7). But there are also new protocols such as OPC-UA which allow to browse available data within the controller. Our goal is to make connectivity of industrial data sources a matter of minutes.

Currently, data sources can be connected using the built-in module `StreamPipes Connect` from the UI. We provide a set of adapters for popular protocols that can be customized, e.g., connection details can be added.

To make it even easier to connect industrial data sources with StreamPipes, we plan to add an OPC-UA browser. This will be part of the entry page of StreamPipes connect and should allow users to enter connection details of an existing OPC-UA server. Afterwards, a new view in the UI shows available data nodes from the server, their status and current value. Users should be able to select values that should be part of a new adapter. Afterwards, a new adapter can be created by reusing the current workflow to create an OPC-UA data source.

This is a really cool project for participants interested in full-stack development who would like to get a deeper understanding of industrial IoT protocols. Have fun!

Tasks

  • [ ] get familiar with the OPC-UA protocol
  • [ ] develop mockups which demonstrate the user workflow
  • [ ] develop a data model for discovering data from OPC-UA
  • [ ] create the backend business logic for the OPC-UA browser
  • [ ] create the frontend views to asynchronously browse data and to create a new adapter
  • [ ] write Junit, Component and E2E tests
  • [ ] what ever comes in your mind 💡 further ideas are always welcome

Relevant Skills

  • interest in Industrial IoT and procotols such as OPC-UA
  • Java development skills
  • Angular/Typescript development skills

Anyways, the most important relevant skill is motivation and readiness to learn during the project!

Learning Material

Reference

Github issue can be found here: https://github.com/apache/streampipes/issues/1390

Name and contact information

  • Mentor: Dominik Riemer (riemer[at]apache.org).
  • Mailing list: (dev[at]streampipes.apache.org)
  • Website: streampipes.apache.org
GSOC-144 Apache EventMesh: EventMesh official website docs by version and demo show Apache EventMesh (incubating)
Apache EventMesh is a fully serverless platform used to build distributed event-driven applications.
Website: https://eventmesh.apache.org
GitHub: https://github.com/apache/incubator-eventmesh
Upstream Issue: https://github.com/apache/incubator-eventmesh/issues/3488
Background
We hope that the community can contribute to the maintenance of documents, including the archiving of Chinese and English content of documents of different release versions, the maintenance of official website documents, the improvement of project quick start documents, feature introduction, etc.
Task
1.Discuss with the mentors what you need to do
2. Learn the details of the Apache EventMesh project
3. Improve and supplement the content of documents on GitHub, maintain official website documents, record eventmesh quick user experience, and feature display videos
Recommended Skills
1.Familiar with MarkDown
2. Familiar with Java\Go
Mentor
Eason Chen, PPMC of Apache EventMesh, https://github.com/qqeasonchen, chenguangsheng@apache.org
Mike Xue, PPMC of Apache EventMesh, https://github.com/xwm1992, mikexue@apache.org
GSOC-140 Apache ShardingSphere: Add ShardingSphere Kafka source connector

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere

Background

The community just added CDC (change data capture) feature recently. Change feed will be published in created network connection after logging in, then it could be consumed.

Since Kafka is popular distributed event streaming platform, it's useful to import change feed into Kafka for later processing.

Task

  1. Familiar with ShardingSphere CDC client usage, create publication and subscribe change feed.
  2. Familiar with Kafka connector development, develop source connector, integrate with ShardingSphere CDC. Persist change feed to Kafka topics properly.
  3. Add unit test and E2E integration test.

Relevant Skills

  1. Java language
  2. Basic knowledge of CDC and Kafka
  3. Maven

References

Local Test Steps

  1. Modify `conf/server.yaml`, uncomment `cdc-server-port: 33071` to enable CDC. (Refer to step 2)
  2. Configure proxy, refer to `Prerequisites` and `Procedure` in build to configure proxy (Newer version could be used too, current stable version is 5.3.1).
  3. Start proxy server, it'll start CDC server too.
  4. Download ShardingSphere source code from https://github.com/apache/shardingsphere , modify and run `org.apache.shardingsphere.data.pipeline.cdc.client.example.Bootstrap`. It'll print `records:` by default in `Bootstrap`.
  5. Execute some ISNERT/UPDATE/DELETE SQLs in proxy to generate change feed, and then check in `Bootstrap` console.

Mentor

Hongsheng Zhong, PMC of Apache ShardingSphere, zhonghongsheng@apache.org

Xinze Guo, Committer of Apache ShardingSphere, azexin@apache.org

GSOC-134 Improving End-to-End Test Infrastructure of Apache StreamPipes

Apache StreamPipes

Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.

Background

StreamPipes has grown significantly over the past few years, with new features and contributors joining the project. However, as the project continues to evolve, e2e test coverage must also be improved to ensure that all features remain functional. Modern frameworks, such as Cypress, make it quite easy and fun to automatically test even complex application functionalities. As StreamPipes approaches its 1.0 release, it is important to improve e2e testing to ensure the robustness of the project and its use in real-world scenarios.

Tasks

  • [ ] Write e2e tests using Cypress to cover most functionalities and user interface components of StreamPipes.
  • [ ] Add more complex testing scenarios to ensure the reliability and robustness of StreamPipes in real-world use cases (e.g. automated tests for version updates)
  • [ ] Add e2e tests for the new Python client to ensure its integration with the main system and its functionalities ([#774 | https://github.com/apache/streampipes/issues/774]])
  • [ ] Document the testing infrastructure and the testing approach to allow for easy maintenance and future contributions.

    ❗ ***Important Note*** ❗

Do not create any account on behalf of Apache StreamPipes in Cypress or using the name of Apache StreamPipes for any account creation. Your mentor will take care of it.

Relevant Skills

  • Familiarity with testing frameworks, such as Cypress or Selenium
  • Experience with TypeScript or Java
  • Basic knowledge of Angular is helpful
  • Familiarity with Docker and containerization is a plus

    Learning Material

References

You can find our corresponding issue on GitHub here

Name and Contact Information

Name: Philipp Zehnder

email: zehnder[at]apache.org

community: dev[at]streampipes.apache.org

website: https://streampipes.apache.org/

GSOC-132 [GSoC] [RocketMQ] The performance tuning of RocketMQ proxy

Apache RocketMQ

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity, and flexible scalability.

Page: https://rocketmq.apache.org
Repo: https://github.com/apache/rocketmq

Background

RocketMQ 5.0 has released a new module called `proxy`, which supports gRPC and remoting protocol. Additionally, it can be deployed in two modes, namely Local and Cluster modes. The performance tuning task will provide contributors with a comprehensive understanding of Apache RocketMQ and its intricate data flow, presenting a unique opportunity for beginners to acquaint themselves with and actively participate in our community.

Task

The task is to tune RocketMQ proxy for optimal performance involves latency and throughput. possess a thorough knowledge of Java implementation and possess the ability to fine-tune Netty, gRPC, the operating system, and RocketMQ itself. We anticipate that the developer responsible for this task will provide a performance report about measurements of both latency and throughput.

Relevant Skills

Basic knowledge of RocketMQ 5.0, Netty, gRPC, and operating system.

Mailing List: dev@rocketmq.apache.org

Mentor
Zhouxiang Zhan, committer of Apache RocketMQ, zhouxzhan@apache.org

GSOC-131 [GSoC][Doris]Page Cache Improvement

Apache Doris
Apache Doris is a real-time analytical database based on MPP architecture. As a unified platform that supports multiple data processing scenarios, it ensures high performance for low-latency and high-throughput queries, allows for easy federated queries on data lakes, and supports various data ingestion methods.
Page: https://doris.apache.org

Github: https://github.com/apache/doris

Background

Apache Doris accelerates high-concurrency queries utilizing page cache, where the decompressed data is stored.
Currently, the page cache in Apache Doris uses a simple LRU algorithm, which reveals a few problems:

  • Hot data will be phased out in large queries
  • The page cache configuration is immutable and does not support GC.

Task

  • Phase One: Identify the impacts on queries when the decompressed data is stored in memory and SSD, respectively, and then determine whether full page cache is required.
  • Phase Two: Improve the cache strategy for Apache Doris based on the results from Phase One.

Learning Material

Page: https://doris.apache.org
Github: https://github.com/apache/doris

Mentor

GSOC-130 Apache ShenYu Gsoc 2023 - Design license scanning function

Background

At present, shenyu needs to manually check whether the license is correct one by one when releasing the version.

Tasks

  1. Discuss with the tutor to complete the requirement design and technical design of the scanning license.
  2. Finished scanning the initial version of the license.
  3. Complete the corresponding test.

Relevant Skills

  1. Familiar with Java.
GSOC-129 [GSOC] [SkyWalking] Unify query planner and executor in BanyanDB

Background

SkyWalking BanyanDB is an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data.

Objectives

  1. Fully unify/merge the query planner and executor for Measure and TopN
  1. Familiar with Go
  2. Have a basic understanding of database query engine
  3. Have an experience of Apache SkyWalking

Mentor

GSOC-126 GSoC: Observability Improvement for RocketMQ Streams

RocketMQ Streams

RocketMQ Streams is a lightweight stream processing framework, application gains the stream processing ability by depending on RocketMQ Streams as an SDK.

Background

Repo of RocketMQ Streams: https://github.com/apache/rocketmq-streams
The architecture document of RocketMQ Streams: RocketMQ Streams examples,

architecture doc

Task

The observability needs to be enhanced in the following aspects:

  • The metric of client/processor/thread/state/rocksdb;
  • The topology of streaming process;
  • Mutli-output of metrics;

This task need you to study the implementation details of RocketMQ Streams streams, and find out the key indicators in the stream processing process. Design and implement a complete set of observability solutions, and finally use it to complete runtime problem diagnosis;

Mentor

nize, Committer of of Apache RocketMQ, karp@apache.org

GSOC-124 [GSOC] [SkyWalking] Pending Task on K8s Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures. This task is about a pending task on K8s.
GSOC-123 Add server indicator if a server is a cache https://github.com/apache/trafficcontrol/issues/7076
GSOC-122 GSoC: Integrate RocketMQ 5.0 client with Spring

Apache RocketMQ

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

Page: https://rocketmq.apache.org
Github: https://github.com/apache/rocketmq

Background

RocketMQ 5.0 client has been released recently, we need to integrate it with Spring.

Related issue: https://github.com/apache/rocketmq-clients/issues/275

Task

  1. Familiar with RocketMQ 5.0 java client usage, you could see more details from https://github.com/apache/rocketmq-clients/tree/master/java and https://rocketmq.apache.org/docs/quickStart/01quickstart
  2. Integrate with Spring.

Relevant Skills

  1. Java language
  2. Basic knowledge of RocketMQ 5.0
  3. Spring

Mentor

Rongtong Jin, PMC of Apache RocketMQ, jinrongtong@apache.org

Yangkun Ai, PMC of Apache RocketMQ, aaronai@apache.org

GSOC-121 [GSoC][Doris] Supports BigQuery/Apache Kudu/Apache Cassandra/Apache Druid in Federated Queries

Apache Doris
Apache Doris is a real-time analytical database based on MPP architecture. As a unified platform that supports multiple data processing scenarios, it ensures high performance for low-latency and high-throughput queries, allows for easy federated queries on data lakes, and supports various data ingestion methods.
Page: https://doris.apache.org
Github: https://github.com/apache/doris

Background

Apache Doris supports acceleration of queries on external data sources to meet users' needs for federated queries and analysis.
Currently, Apache Doris supports multiple external catalogs including those from Hive, Iceberg, Hudi, and JDBC. Developers can connect more data sources to Apache Doris based on a unified framework.

Objective

Task
Phase One:

  • Get familiar with the Multi-Catalog structure of Apache Doris, including the metadata synchronization mechanism in FE and the data reading mechanism of BE.
  • Investigate how metadata should be acquired and how data access works regarding the picked data source(s); produce the corresponding design documentation.

Phase Two:

  • Develop connections to the picked data source(s) and implement access to metadata and data.

Learning Material

Page: https://doris.apache.org
Github: https://github.com/apache/doris

Mentor

GSOC-120 [GSoC][Doris]Dictionary Encoding Acceleration

Apache Doris
Apache Doris is a real-time analytical database based on MPP architecture. As a unified platform that supports multiple data processing scenarios, it ensures high performance for low-latency and high-throughput queries, allows for easy federated queries on data lakes, and supports various data ingestion methods.
Page: https://doris.apache.org

Github: https://github.com/apache/doris

Background

In Apache Doris, dictionary encoding is performed during data writing and compaction. Dictionary encoding will be implemented on string data types by default. The dictionary size of a column for one segment is 1M at most. The dictionary encoding technology accelerates strings during queries, converting them into INT, for example.

Task

  • Phase One: Get familiar with the implementation of Apache Doris dictionary encoding; learning how Apache Doris dictionary encoding accelerates queries.
  • Phase Two: Evaluate the effectiveness of full dictionary encoding and figure out how to optimize memory in such a case.

Learning Material

Page: https://doris.apache.org
Github: https://github.com/apache/doris

Mentor

GSOC-118 [GSOC][SkyWalking] Add Terraform provider for Apache SkyWalking Now the deployment methods for SkyWalking are limited, we only have Helm Chart for users to deploy in Kubernetes, other users that are not using Kubernetes have to do all the house keeping stuffs to set up SkyWalking on, for example, VM.
This issue aims to add a Terraform provider, so that users can conveniently spin up a cluster for demonstration or testing, we should evolve the provider and allow users to customize as their need and finally users can use this in their production environment.
In this task, we will mainly focus on the support for AWS. In the Terraform provider, users need to provide their access key / secret key, and the provider does the rest stuffs: create VMs, create database/OpenSearch or RDS, download SkyWalking tars, configure the SkyWalking, and start the SkyWalking components (OAP/UI), create public IPs/domain name, etc.
GSOC-117 GSoC: Make RocketMQ support higher versions of Java

Apache RocketMQ

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

Page: https://rocketmq.apache.org
Github: https://github.com/apache/rocketmq

Background

RocketMQ is a widely used message middleware system in the Java community, which mainly supports Java8. As Java has evolved many new features and improvements have been added to the language and the Java Virtual Machine (JVM). However, RocketMQ still lacks compatibility with the latest Java versions, preventing users from taking advantage of new features and performance improvements. Therefore, we are seeking community support to upgrade RocketMQ to support higher versions of Java and enable the use of new features and JVM parameters.

Task

We aim to update the RocketMQ codebase to support newer versions of Java in a cross-compile manner. The goal is to enable RocketMQ to work with Java17, while maintaining backward compatibility with previous versions of Java. This will involve identifying and updating any dependencies that need to be changed to support the new Java versions, as well as testing and verifying that the new version of RocketMQ works correctly. With these updates, users will be able to take advantage of the latest Java features and performance improvements. We hope that the community can come together to support this task and make RocketMQ a more versatile and powerful middleware system.

Relevant Skills

  1. Java language
  2. Having a good understanding of the new features in higher versions of Java, particularly LTS versions.

Mentor

Yangkun Ai, PMC of Apache RocketMQ, aaronai@apache.org

GSOC-115 RocketMQ DLedger Controller Performance Optimization

Apache RocketMQ

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity, and flexible scalability.
Page: https://rocketmq.apache.org
Repo: https://github.com/apache/rocketmq

Background

RocketMQ 5.0 introduced a new component, the controller, which controls the high availability master-slave switch in multi-replica scenarios. It uses the DLedger Raft library as a consensus replication state machine for metadata. As a completely independent component, it can run normally in some scenarios, but in large-scale clusters, it is necessary to maintain a large number of broker groups, which is a great challenge for operational capabilities and resource waste. When dealing with a large number of Broker groups, we need to optimize performance in large-scale scenarios, leveraging the high-performance writing of DLedger itself and performing some optimization for the current Controller architecture.

Task

1. Polish the usage of DLedger

Currently, on the Controller side, a task queue single thread is used for requesting reads and writes to DLedger, that is, only one read/write request can be processed at a time. However, DLedger itself implements many optimizations for multi-client reads and writes and can ensure linear consistency reading. However, now all read and write processing is performed using a single logical DLedger client, which will become a serious performance bottleneck in large-scale scenarios.

2. Optimization of DLedger features usage

DLedger itself can implement many optimizations, such as ReadIndex read and FollowerRead read. After implementation, we can fully leverage the performance of reads. Currently, all Broker nodes communicate with the Leader node of the Controller. In large-scale scenarios, this will cause the requests of each Controller group to be concentrated on the Leader node, and the other Follower nodes will not share the request processing of the Leader, which will cause single-point performance bottlenecks for the Leader.

3. Full asynchronous + parallel processing

Currently, DLedger itself is fully asynchronous, but on the Controller side, all requests for the DLedger side are synchronized, and many Controller-side operations are performed synchronously in a single thread, such as heartbeat checks and other timed tasks. In large-scale scenarios, the logic of these single-threaded synchronous operations will block a large number of requests from Broker-side, so asynchronous + parallel processing can be used for optimization.

4. Correctness testing and performance testing

After completing the above optimizations, it is necessary to conduct correctness testing on the new version and use distributed chaos testing frameworks such as OpenChaos to verify correct operation under fault scenarios such as network partition and random crashes.
After completing the correctness testing, a detailed performance testing report can be produced by comparing the new and old versions.

Skills Required

  • Strong interest in message middleware and distributed storage systems
  • Proficient in Java development
  • In-depth understanding of distributed consensus algorithms
  • In-depth understanding of the high-availability module of RockeetMQ and the DLedger library
  • Understanding of distributed chaos testing and performance testing.
GSOC-114 [GSOC] [SkyWalking] Python Agent Performance Enhancement Plan

Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures. This task is about enhancing Python agent performance, the tracking issue can be seen here -< https://github.com/apache/skywalking/issues/10408

Mentor

GSOC-111 [GSoC][Beam] An IntelliJ plugin to develop Apache Beam pipelines and the Apache Beam SDKs Beam library developers and Beam users would appreciate this : )
This project involves prototyping a few different solutions, so it will be large.
GSOC-110 GSoC: Support Logging exporter for metrics in RocketMQ

Apache RocketMQ

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

Page: https://rocketmq.apache.org
Github: https://github.com/apache/rocketmq

Background

RocketMQ 5.0 supports metrics based on OpenTelemetry. However, typically metrics data needs to be collected by an OpenTelemetry Collector, in which case the OTLP metrics exporter is used. If there is no collector-based component, it is also possible to print the metrics data directly to logs, which makes troubleshooting problems more convenient.

https://github.com/apache/rocketmq/issues/5678

Relevant Skills

  1. Java language
  2. Basic knowledge of CNCF OpenTelemetry

Mentor

Yangkun Ai, PMC of Apache RocketMQ, aaronai@apache.org

GSOC-108 GSoC: Implement python client for RocketMQ 5.0

Apache RocketMQ

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

Page: https://rocketmq.apache.org

Background

RocketMQ 5.0 has released various language clients including Java, CPP, and Golang, to cover all major programming languages, a Python client needs to be implemented.

Related Repo: https://github.com/apache/rocketmq-clients

Task

The developer is required to be familiar with the Java implementation and capable of developing a Python client, while ensuring consistent functionality and semantics.

Relevant Skills
Python language
Basic knowledge of RocketMQ 5.0

Mentor

Yangkun Ai, PMC of Apache RocketMQ, aaronai@apache.org

GSOC-107 [GSOC] [SkyWalking] AIOps Log clustering with Flink (Algorithm Optimization)

Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures. This year we will proceed on log clustering implementation with a revised architecture and this task will require student to focus on algorithm optimiztion for the clustering technique.

Mentor

GSOC-106 [GSOC] [SkyWalking] AIOps Log clustering with Flink (Flink Integration)

Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures. This year we will proceed on log clustering implementation with a revised architecture and this task will require student to focus on Flink and its integration with SkyWalking OAP.

Mentor

GSOC-105 RocketMQ TieredStore Integration with High Availability Architecture

Apache RocketMQ{}

Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.

Page: https://rocketmq.apache.org

Background

With the official release of RocketMQ 5.1.0, tiered storage has arrived as a new independent module in the Technical Preview milestone. This allows users to unload messages from local disks to other cheaper storage, extending message retention time at a lower cost.

Reference RIP-57: https://github.com/apache/rocketmq/wiki/RIP-57-Tiered-storage-for-RocketMQ

In addition, RocketMQ introduced a new high availability architecture in version 5.0.

Reference RIP-44: https://github.com/apache/rocketmq/wiki/RIP-44-Support-DLedger-Controller

However, currently RocketMQ tiered storage only supports single replicas.

Task

Currently, tiered storage only supports single replicas, and there are still the following issues in the integration with the high availability architecture:

  • Metadata synchronization: how to reliably synchronize metadata between master and slave nodes.
  • Disallowing message uploads beyond the confirm offset: to avoid message rollback, the maximum uploaded offset cannot exceed the confirm offset.
  • Starting multi-tier storage upload when the slave changes to master, and stopping tiered storage upload when the master becomes the slave: only the master node has write and delete permissions, and after the slave node is promoted, it needs to quickly resume tiered storage breakpoint resumption.
  • Design of slave pull protocol: how a newly launched empty slave can properly synchronize data through the tiered storage architecture. (If synchronization is performed based on the first or last file, resumption of breakpoints may not be possible when switching again).

So you need to provide a complete plan to solve the above issues and ultimately complete the integration of tiered storage and high availability architecture, while verifying it through the existing tiered storage file version and OpenChaos testing.

Relevant Skills

  • Interest in messaging middleware and distributed storage systems
  • Java development skills
  • Having a good understanding of RocketMQ tiered storage and high availability architecture
GSOC-103 [GSoC][Beam] Build out Beam Machine Learning Use Cases Today, you can do all sorts of Machine Learning using Apache Beam (https://beam.apache.org/documentation/ml/overview/).

Many of our users, however, have a hard time getting started with ML and understanding how Beam can be applied to their day to day work. The goal of this project is to build out a series of Beam pipelines as Jupyter Notebooks demonstrating real world ML use cases, from NLP to image recognition to using large language models. As you go, there may be bugs or friction points as well which will provide opportunities to contribute back to Beam's core ML libraries.
Mentor for this will be Danny McCormick
GSOC-100 GSOC: Varnish Cache support in Apache Traffic Control

Background
Apache Traffic Control is a Content Delivery Network (CDN) control plane for large scale content distribution.

Traffic Control currently requires Apache Traffic Server as the underlying cache. Help us expand the scope by integrating with the very popular Varnish Cache.

There are multiple aspects to this project:

  • Configuration Generation: Write software to build Varnish configuration files (VCL). This code will be implemented in our Traffic Ops and cache client side utilities, both written in Go.
  • Health Monitoring: Implement monitoring of the Varnish cache health and performance. This code will run both in the Traffic Monitor component and within Varnish. Traffic Monitor is written in Go and Varnish is written in C.
  • Testing: Adding automated tests for new code

Skills:

  • Proficiency in Go is required
  • A basic knowledge of HTTP and caching is preferred, but not required for this project.
GSOC-98 Apache ShenYu Gsoc 2023 - Design and implement shenyu ingress-controller in k8s

Background

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance. Currently, ShenYu has good usability and performance in microservice scenarios. However, ShenYu's support for Kubernetes is still relatively weak.

Tasks

1. Discuss with mentors, and complete the requirements design and technical design of shenyu-ingress-controller.
2. Complete the initial version of shenyu-ingress-controller, implement the reconcile of k8s ingress api, and make ShenYu as the ingress gateway of k8s.
3. Complete the ci test of shenyu-ingress-controller, verify the correctness of the code.

Relevant Skills

1. Know the use of Apache ShenYu
2. Familiar with Java and Golang
3. Familiar with Kubernetes and can use java or golang to develop Kubernetes Controller

Description

Issues : https://github.com/apache/shenyu/issues/4438
website : https://shenyu.apache.org/

GSOC-93 CloudStack GSoC 2023 - Extend Import-Export Instances to the KVM Hypervisor

Github issue: https://github.com/apache/cloudstack/issues/7127

Description:

The Import-Export functionality is only allowed for the Vmware hypervisor. The functionality is developed within a VM ingestion framework that allows the extension to other hypervisors. The Import-Export functionality consists on few APIs and the UI to interact with them:

  • listUnmanagedInstances: Lists unmanaged virtual machines (not existing in CloudStack but existing on the hypervisor side)
  • importUnmanagedInstance: Import an unmanaged VM into CloudStack (this implies populating the database with the corresponding data)
  • unmanageVirtualMachine: Make CloudStack forget a VM but do not remove it on the hypervisor side

The complexity on KVM should be parsing the existing XML domains into different resources and map them in CloudStack to populate the database correctly.

GSOC-61 Apache ShenYu Gsoc 2023 - Support for Kubernetes Service Discovery Background
Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance. Currently, ShenYu has good usability and performance in microservice scenarios. However, ShenYu's support for Kubernetes is still relatively weak.
Tasks
1. Support the registration of microservices deployed in K8s Pod to shenyu-admin and use K8s as the register center.
2. Discuss with mentors, and complete the requirements design and technical design of Shenyu K8s Register Center.
3. Complete the initial version of Shenyu K8s Register Center.
4. Complete the CI test of Shenyu K8s Register Center, verify the correctness of the code.
5. Write the necessary documentation, deployment guides, and instructions for users to connect microservices running inside the K8s Pod to ShenYu
Relevant Skills
1. Know the use of Apache ShenYu, especially the register center
2. Familiar with Java and Golang
3. Familiar with Kubernetes and can use Java or Golang to develop
GSOC-42 Apache EventMesh: Integrate eventmesh runtime on Kubernetes Apache EventMesh (incubating)
Apache EventMesh is a fully serverless platform used to build distributed event-driven applications.
Website: https://eventmesh.apache.org
GitHub: https://github.com/apache/incubator-eventmesh
Upstream Issue: https://github.com/apache/incubator-eventmesh/issues/3327
Background
Currently, EventMesh has good usability in microservice scenarios. However, EventMesh's support for Kubernetes is still relatively weak.We hope the community can contribute EventMesh integration with the k8s.
Task
1.Discuss with the mentors your implementation idea
2. Learn the details of the Apache EventMesh project
3. Integrate EventMesh with the k8s
Recommended Skills
1.Familiar with Java
2.Familiar with Kubernetes
Mentor
Eason Chen, PPMC of Apache EventMesh, https://github.com/qqeasonchen, chenguangsheng@apache.org
Mike Xue, PPMC of Apache EventMesh, https://github.com/xwm1992, mikexue@apache.org
GSOC-33 Apache ShenYu Gsoc 2023 - Shenyu-Admin Internationalization

Background

Shenyu is a native API gateway for service proxy, protocol translation and API governance. It can manage and maintain the API through Shenyu-admin, and support internationalization in Chinese and English. Unfortunately, Shenyu-admin is only internationalized on the front end. The message prompt returned by the back-end interface is still in English. Therefore, we need to implement internationalization support for the back-end interface.This will lay a good foundation for shenyu to move towards more language support.

Relevant skills

  • Related skills spring resources
  • Spring Internationalization
  • Front-end react framework

API reference

``` java.util.Locale; org.springframework.context.MessageSource; org.springframework.context.support.ResourceBundleMessageSource; ```

Interface effect example

``` ## zh request example POST http://localhost:9095/plugin Content-Type: application/json Location: cn-zh X-Access-Token: xxx { "name": "test-create-plugin", "role": "test-create-plugin", "enabled": true, "sort": 100 } Respone { "code": 600, "message": "未登录" }

en request example

POST http://localhost:9095/plugin Content-Type: application/json Location: en X-Access-Token: xxx { “name”: “test-create-plugin”, “role”: “test-create-plugin”, “enabled”: true, “sort”: 100 } Respone { “code”: 600, “message”: “token is error” }

<h3 id="task-list-2">Task List</h3>
<ul>
<li>The task discussed with the tutor how to achieve the internationalization of shenyu-admin background</li>
<li>Some prompt message translation</li>
<li>Get through the internationalization of front-end, obtain the client region information through http protocol, support the language of the corresponding region.</li>
<li>Leave the extension of other multi-language internationalization support interface, so as to facilitate the localization transformation of subsequent users.</li>
</ul></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/GSOC-32">GSOC-32</a></td>
<td><a href="https://issues.apache.org/jira/browse/GSOC-32">Apache EventMesh: Support source/sink connectors on EventMesh</a></td>
<td></td>
<td></td>
<td></td>
<td><strong>Apache EventMesh (incubating)</strong><br>Apache EventMesh is a fully serverless platform used to build distributed event-driven applications.<br>Website: <a href="https://eventmesh.apache.org/">https://eventmesh.apache.org</a><br>GitHub: <a href="https://github.com/apache/incubator-eventmesh">https://github.com/apache/incubator-eventmesh</a><br>Upstream Issue: <a href="https://github.com/apache/incubator-eventmesh/issues/3492">https://github.com/apache/incubator-eventmesh/issues/3492</a><br><strong>Background</strong><br>Through eventmesh's source/sink connector, we can connect data to heterogeneous data storage, we hope that the community can provide source/sink connector capabilities, such as connecting rocketmq data to different rocketmq clusters or kafka clusters<br><strong>Task</strong><br>1. Discuss with the mentors what you need to do<br>2. Learn the details of the Apache EventMesh project<br>3. Implement one of the source/sink connector based on the above background<br><strong>Recommended Skills</strong><br>1. Familiar with Java<br>2. Familiar with MQ is better<br><strong>Mentor</strong><br>Eason Chen, PPMC of Apache EventMesh, <a href="https://github.com/qqeasonchen">https://github.com/qqeasonchen</a>, chenguangsheng@apache.org<br>Mike Xue, PPMC of Apache EventMesh, <a href="https://github.com/xwm1992">https://github.com/xwm1992</a>, mikexue@apache.org</td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/GSOC-31">GSOC-31</a></td>
<td><a href="https://issues.apache.org/jira/browse/GSOC-31">[GSoC] RocketMQ TieredStore Integration with HDFS</a></td>
<td></td>
<td></td>
<td></td>
<td><h3 id="gsoc-rocketmq-tieredstore-integration-with-hdfs">[GSoC] RocketMQ TieredStore Integration with HDFS</h3>
<p>Github Issue: <a href="https://github.com/apache/rocketmq/issues/6282">https://github.com/apache/rocketmq/issues/6282</a></p>
<h4 id="apache-rocketmq-and-hdfs">Apache RocketMQ and HDFS</h4>
<ul>
<li>Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.</li>
</ul>
<ul>
<li>Hadoop Distributed File System (HDFS) is a distributed file system designed to store and manage large data sets across multiple servers or clusters. HDFS provides a reliable, scalable, and fault-tolerant platform for storing and accessing data that can be accessed by a variety of applications running on the hadoop cluster.</li>
</ul>
<h4 id="background-27">Background</h4>
<p>High-speed storage media, such as solid-state drives (SSDs), are typically more expensive than traditional hard disk drives (HDDs). To minimize storage costs, the local data disk size of a rocketmq broker is often limited. HDFS can store large amounts of data at a lower cost, it has better support for storing and retrieving data sequentially rather than randomly. In order to preserve message data over a long period or facilitate message export, the RocketMQ project previously introduced a tiered storage plugin. Now it is necessary to implement a storage plugin to save data on hdfs.</p>
<h4 id="relevant-skills-19">Relevant Skills</h4>
<ul>
<li>Interest in messging middleware and distributed storage system</li>
</ul>
<ul>
<li>Java development skills</li>
</ul>
<ul>
<li>Having a good understanding of rocketmq and hdfs models</li>
</ul>
<p>Anyways, the most important relevant skill is motivation and readiness to learn during the project!</p>
<h4 id="tasks-4">Tasks</h4>
<ul>
<li>understand the basic concepts and principles in distributed systems</li>
</ul>
<ul>
<li>provide related design documents</li>
</ul>
<ul>
<li>develop one that uses hdfs as the backend storage plugin to store rocketmq message data</li>
</ul>
<ul>
<li>write effective unit test code</li>
</ul>
<ul>
<li>*suggest improvements to the tiered storage interface</li>
</ul>
<ul>
<li>*what ever comes in your mind further ideas are always welcome</li>
</ul>
<h4 id="learning-material-6">Learning Material</h4>
<ul>
<li>RocketMQ HomePage (<a href="https://rocketmq.apache.org/">https://rocketmq.apache.org</a>) <strong>Github</strong>: <a href="https://github.com/apache/rocketmq">https://github.com/apache/rocketmq</a></li>
</ul>
<ul>
<li>RocketMQ Tiered Storage Design (<a href="https://github.com/apache/rocketmq/wiki/RIP-57-Tiered-storage-for-RocketMQ">https://github.com/apache/rocketmq/wiki/RIP-57-Tiered-storage-for-RocketMQ</a>)</li>
</ul>
<ul>
<li>HDFS HomePage (<a href="https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html</a>)</li>
</ul>
<h4 id="name-and-contact-information-3">Name and contact information</h4>
<ul>
<li>Mentor: Zhimin Li, Apache RocketMQ Committer, <a href="mailto:lizhimin@apache.org">lizhimin@apache.org<sup></sup></a></li>
</ul>
<ul>
<li>Mailing List: <a href="mailto:dev@rocketmq.apache.org">dev@rocketmq.apache.org<sup></sup></a></li>
</ul>
<ul>
<li>Website: <a href="https://rocketmq.apache.org/">https://rocketmq.apache.org/</a> and <a href="https://hadoop.apache.org/">https://hadoop.apache.org/</a></li>
</ul></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/GORA-709">GORA-709</a></td>
<td><a href="https://issues.apache.org/jira/browse/GORA-709">GSoC/Outreachy Project Idea</a></td>
<td></td>
<td></td>
<td></td>
<td>I would like to propose a project idea suitable for GSoC and Outreachy comprises of following sub tasks.<br>1. Complete rest of unfinished work on Geode module - <a href="https://issues.apache.org/jira/browse/GORA-698">https://issues.apache.org/jira/browse/GORA-698</a><br>2. Upgrade Hadoop version - <a href="https://issues.apache.org/jira/browse/GORA-537">https://issues.apache.org/jira/browse/GORA-537</a><br>We could scope out the project adding / removing sub tasks based on the available capacity of the student and project.</td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/GORA-708">GORA-708</a></td>
<td><a href="https://issues.apache.org/jira/browse/GORA-708">GSoC/Outreachy Project Idea</a></td>
<td></td>
<td></td>
<td></td>
<td>I would like to propose a project idea suitable for GSoC and Outreachy comprises of following sub tasks.<br>1. Complete rest of unfinished work on ArangoDB module - <a href="https://issues.apache.org/jira/browse/GORA-650">https://issues.apache.org/jira/browse/GORA-650</a><br>2. Upgrade HBase driver - <a href="https://issues.apache.org/jira/browse/GORA-706">https://issues.apache.org/jira/browse/GORA-706</a><br>3. Upgrade Hive driver - <a href="https://issues.apache.org/jira/browse/GORA-707">https://issues.apache.org/jira/browse/GORA-707</a><br>We could scope out the project adding / removing sub tasks based on the available capacity of the student and project.</td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/FINERACT-1586">FINERACT-1586</a></td>
<td><a href="https://issues.apache.org/jira/browse/FINERACT-1586">Reduce Boilerplate Code by Introducing lombok to Reduce getters/setters and Mapstruct to map REST DTO to Entity Objects</a></td>
<td></td>
<td></td>
<td></td>
<td><p>Lombok could help us to not only reduce a large amount of code, but also to fix a couple of inconsistencies in the code base:</p>
<ul>
<li>getters/setters with non-standard characters (e. g. underscores)</li>
</ul>
<ul>
<li>getters/setters with typos</li>
</ul>
<p>The layered architecture of Fineract requires mapping between REST DTO classes and internal entity classes. The current code base contains various strategies to achieve this:</p>
<ul>
<li>private functions</li>
</ul>
<ul>
<li>static functions</li>
</ul>
<ul>
<li>mapping classes</li>
</ul>
<p>All of these approaches are very manual (and error prone) and difficult to maintain. Mapstruct can help here:</p>
<ul>
<li>throw errors at compile time (missing new attributes, type changes etc.)</li>
</ul>
<ul>
<li>one common concept (easier to understand)</li>
</ul>
<ul>
<li>reduce manually maintained code and replace mostly generated code</li>
</ul>
<p>Challenges:</p>
<ul>
<li>maintain immutability (especially in DTO classes)</li>
</ul>
<ul>
<li>should we fluent builder pattern?</li>
</ul>
<ul>
<li>backwards compatibility</li>
</ul>
<ul>
<li>these improvements cannot be introduced as one pull request, but have to be split up at least at the “module” level (clients, loans, accounts etc.). This would result in approximately 30 pull requests; if we split up Lombok and Mapstruct then it would be 30 PRs each (=60); we would need this fine grained approach to make a transition as painless as possible</li>
</ul>
<ul>
<li>some classes are maybe beyond repair (e. g. Loan.java with 6000 lines of code, the smaller part getters/setters and a long list of utility/business logic functions)</li>
</ul></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-135">DUBBO-135</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-135">Dubbo GSoC - Pixiu supports gRPC/dubbo protocol with WASM plug-in</a></td>
<td></td>
<td></td>
<td></td>
<td>Pixiu acts as a gateway, forwarding traffic to various services.<br>Pixiu needs to support communication between different applications on the browser, and WASM needs to be supported on the browser. Currently, it only supports the HTTP protocol.<br>This project needs to complete the communication protocol below WASM (gRPC is preferred)<br>1. Support gRPC protocol<br>2. Support dubbo protocol<br>The front end calls gRPC reference <a href="https://github.com/grpc/grpc-web">https://github.com/grpc/grpc-web</a></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-134">DUBBO-134</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-134">Dubbo GSoC 2023 - Automatically configure pixiu as istio ingress gateway</a></td>
<td></td>
<td></td>
<td></td>
<td>In the istio mesh environment, the public dubbo/dubbo go provider can be exposed outside the cluster through the http/https protocol through the istio ingress gateway. This requires the ingress gateway to complete the conversion from http to dubbo protocol, which is the main scenario of pixiu; this project Need to complete:<br>1. Customize pixiu, which can be used as an istio ingress gateway, proxy http/https requests and convert them into dubbo requests;<br>2. The gateway supports basic user authentication methods.<br>Basic reference: <a href="https://istio.io/latest/blog/2019/custom-ingress-gateway/">https://istio.io/latest/blog/2019/custom-ingress-gateway/</a><br><a href="https://cloud.ibm.com/docs/containers?topic=containers-istio-custom-gateway">https://cloud.ibm.com/docs/containers?topic=containers-istio-custom-gateway</a></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-133">DUBBO-133</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-133">Dubbo GSoC 2023 - Dubbo3 Python HTTP/2 RPC Protocol Implementation</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-132">DUBBO-132</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-132">Dubbo GSoC 2023 - Rust Cluster Feature Implementation and Stability Improvement.</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-131">DUBBO-131</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-131">Dubbo GSoC 2023 - Dubbo3 Node.js HTTP/2 RPC Protocol Implementation</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-130">DUBBO-130</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-130">Dubbo GSoC 2023 - Development of Dubbo Admin Dashboard UI Pages</a></td>
<td></td>
<td></td>
<td></td>
<td>In charge of the maintenance of the development of the UI pages of the whole Dubbo Admin project.</td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-129">DUBBO-129</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-129">Dubbo GSoC 2023 - Admin Control Plane</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-128">DUBBO-128</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-128">Dubbo GSoC 2023 - Go Observability Improvement</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-127">DUBBO-127</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-127">Dubbo GSoC 2023 - Improve usability of Dubbo-go project</a></td>
<td></td>
<td></td>
<td></td>
<td>Including but not limited to programming patterns, configuration, apis, documentation and demos.</td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-126">DUBBO-126</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-126">Dubbo GSoC 2023 - Go Web Protocol and Programming Support</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-125">DUBBO-125</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-125">Dubbo GSoC 2023 - Go HTTP1&amp;2 RPC Protocol Support</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-124">DUBBO-124</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-124">Dubbo GSoC 2023 - Go Security</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-123">DUBBO-123</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-123">Dubbo GSoC 2023 - Go Traffic Management</a></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-122">DUBBO-122</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-122">Dubbo GSoC 2023 - Service Deployer</a></td>
<td></td>
<td></td>
<td></td>
<td>For a large number of monolithic applications, problems such as performance will be encountered during large-scale deployment. For interface-oriented programming languages, Dubbo provides the capability of RPC remote calls, and we can help applications decouple through interfaces. Therefore, we can provide a deployer to help users realize the decoupling and splitting of microservices during deployment, and quickly provide performance optimization capabilities.</td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-121">DUBBO-121</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-121">Dubbo GSoC 2023 - API manager</a></td>
<td></td>
<td></td>
<td></td>
<td>Since Dubbo runs on a distributed architecture, it naturally has the problem of difficult API interface definition management. It is often difficult for us to know which interface is running in the production environment. So we can provide an API-defined reporting platform, and even a management platform. This platform can automatically collect all APIs of the cluster, or can be directly defined by the user, and then unified distribution management is carried out through a mechanism similar to git and maven package management.</td>
</tr>
<tr>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-120">DUBBO-120</a></td>
<td><a href="https://issues.apache.org/jira/browse/DUBBO-120">Dubbo GSoC 2023 - JSON compatibility check</a></td>
<td></td>
<td></td>
<td></td>
<td>Dubbo currently supports a large number of Java language features through hessian under the Java SDK, such as generics, interfaces, etc. These capabilities will not be compatible when calling across systems. Therefore, Dubbo needs to provide the ability to detect the interface definition and determine whether the interface published by the user can be described by native json.</td>
</tr>
</tbody>
</table>

Showing 100 out of [132 issues](https://issues.apache.org/jira/secure/IssueNavigator.jspa)