Self-supervised learning (SSL) has emerged as a desirable paradigm in computer vision due to the inability of supervised models to learn representations that can generalize in domains with limited labels. The recent popularity of SSL has led to the development of several models that make use of diverse training strategies, architectures, and data augmentation policies with no existing unified framework to study or assess their effectiveness in transfer learning. We propose a data-driven geometric strategy to analyze different SSL models using local neighborhoods in the feature space induced by each. Unlike existing approaches that consider mathematical approximations of the parameters, individual components, or optimization landscape, our work aims to explore the geometric properties of the representation manifolds learned by SSL models. Our proposed manifold graph metrics (MGMs) provide insights into the geometric similarities and differences between available SSL models, their invariances with respect to specific augmentations, and their performances on transfer learning tasks. Our key findings are two fold: (i) contrary to popular belief, the geometry of SSL models is not tied to its training paradigm (contrastive, non-contrastive, and cluster-based); (ii) we can predict the transfer learning capability for a specific model based on the geometric properties of its semantic and augmentation manifolds.

Sustainable use of biological resources is very important as over exploitation on the long run may lead to stock depletion, which in turn may threaten biodiversity. The Chesapeake Bay is an extremely complex ecosystem, and sustainable harvesting of its fisheries is essential both for the ecosystem's biodiversity and economic prosperity of the area. Here, we use ecosystem based mathematical modeling to study the population dynamics with harvesting of two key fishes in the Chesapeake Bay, the Atlantic Menhaden (Brevoortia tyrannus) as a prey and the Striped Bass (Morone saxatilis) as a predator. We start by fitting the generalized Lotka-Volterra model to actual time series abundance data of the two species obtained from fisheries in the Bay. We derive conditions for the existence of the bio-economic equilibrium and investigate the stability and the resilience of the biological system. We study the maximum sustainable yield, maximum economic yield, and resilience maximizing yield policies and their effects on the fisheries long term sustainability, particularly with respect to the menhaden-bass population dynamics. This study may be used by policy-makers to balance the economic and ecological harvesting goals while managing the populations of Atlantic menhaden and striped bass in the Chesapeake Bay fisheries.

Deep learning-based algorithms have seen a massive popularity in different areas of remote sensing image analysis over the past decade. Recently, transformers-based architectures, originally introduced in natural language processing, have pervaded computer vision field where the self-attention mechanism has been utilized as a replacement to the popular convolution operator for capturing long-range dependencies. Inspired by recent advances in computer vision, remote sensing community has also witnessed an increased exploration of vision transformers for a diverse set of tasks. Although a number of surveys have focused on transformers in computer vision in general, to the best of our knowledge we are the first to present a systematic review of recent advances based on transformers in remote sensing. Our survey covers more than 60 recent transformers-based methods for different remote sensing problems in sub-areas of remote sensing: very high-resolution (VHR), hyperspectral (HSI) and synthetic aperture radar (SAR) imagery. We conclude the survey by discussing different challenges and open issues of transformers in remote sensing. Additionally, we intend to frequently update and maintain the latest transformers in remote sensing papers with their respective code at: https://github.com/VIROBO-15/Transformer-in-Remote-Sensing

The global rise in anthropogenic reactive nitrogen (N) and the negative impacts of N deposition on terrestrial plant diversity are well-documented. The R* theory of resource competition predicts reversible decreases in plant diversity in response to N loading. However, empirical evidence for the reversibility of N-induced biodiversity loss is mixed. In a long-term N-enrichment experiment in Minnesota, a low-diversity state that emerged during N addition has persisted for decades after additions ceased. Hypothesized mechanisms preventing recovery of biodiversity include nutrient recycling, insufficient external seed supply, and litter inhibition of plant growth. Here we present an ODE model that unifies these mechanisms, produces bistability at intermediate N inputs, and qualitatively matches the observed hysteresis at Cedar Creek. Key features of the model, including native species' growth advantage in low-N conditions and limitation by litter accumulation, generalize from Cedar Creek to North American grasslands. Our results suggest that effective biodiversity restoration in these systems may require management beyond reducing N inputs, such as burning, grazing, haying, and seed additions. By coupling resource competition with an additional inter-specific inhibitory process, the model also illustrates a general mechanism for bistability and hysteresis that may occur in multiple ecosystem types.

Generative self-supervised learning (SSL), especially masked autoencoders, has become one of the most exciting learning paradigms and has shown great potential in handling graph data. However, real-world graphs are always heterogeneous, which poses three critical challenges that existing methods ignore: 1) how to capture complex graph structure? 2) how to incorporate various node attributes? and 3) how to encode different node positions? In light of this, we study the problem of generative SSL on heterogeneous graphs and propose HGMAE, a novel heterogeneous graph masked autoencoder model to address these challenges. HGMAE captures comprehensive graph information via two innovative masking techniques and three unique training strategies. In particular, we first develop metapath masking and adaptive attribute masking with dynamic mask rate to enable effective and stable learning on heterogeneous graphs. We then design several training strategies including metapath-based edge reconstruction to adopt complex structural information, target attribute restoration to incorporate various node attributes, and positional feature prediction to encode node positional information. Extensive experiments demonstrate that HGMAE outperforms both contrastive and generative state-of-the-art baselines on several tasks across multiple datasets. Codes are available at https://github.com/meettyj/HGMAE.

Rapid urbanization is a major cause of habitat and biodiversity loss and human-animal conflict. While urbanization is inevitable, we need to develop a good understanding of the urban ecosystem and the urban-adapted species in order to ensure sustainable cities for our future. Scavengers play a major role in urban ecosystems, and often, urban adaptation involves a shift towards scavenging behaviour in wild animals. We carried out an experiment at different sites in the state of West Bengal, India, to identify the scavenging guild within urban habitats, in response to human provided food. Our study revealed a total of 17 different vertebrate species were identified across sites over 498 sessions of observations. We carried out network analysis to understand the dynamics of the system, and found that the free-ranging dog and common mynah were key species within the scavenging networks. This study revealed the complexity of scavenging networks within human-dominated habitats.

LAMDA-SSL is open-sourced on GitHub and its detailed usage documentation is available at https://ygzwqzd.github.io/LAMDA-SSL/. This documentation introduces LAMDA-SSL in detail from various aspects and can be divided into four parts. The first part introduces the design idea, features and functions of LAMDA-SSL. The second part shows the usage of LAMDA-SSL by abundant examples in detail. The third part introduces all algorithms implemented by LAMDA-SSL to help users quickly understand and choose SSL algorithms. The fourth part shows the APIs of LAMDA-SSL. This detailed documentation greatly reduces the cost of familiarizing users with LAMDA-SSL toolkit and SSL algorithms.

The connection between the design and delivery of health care services using information technology is known as health informatics. It involves data usage, validation, and transfer of an integrated medical analysis using neural networks of multi-layer deep learning techniques to analyze complex data. For instance, Google incorporated ''DeepMind'' health mobile tool that integrates \& leverage medical data needed to enhance professional healthcare delivery to patients. Moorfield Eye Hospital London introduced DeepMind Research Algorithms with dozens of retinal scans attributes while DeepMind UCL handled the identification of cancerous tissues using CT \& MRI Scan tools. Atomise analyzed drugs and chemicals with Deep Learning Neural Networks to identify accurate pre-clinical prescriptions. Health informatics makes medical care intelligent, interactive, cost-effective, and accessible; especially with DL application tools for detecting the actual cause of diseases. The extensive use of neural network tools leads to the expansion of different medical disciplines which mitigates data complexity and enhances 3-4D overlap images using target point label data detectors that support data augmentation, un-semi-supervised learning, multi-modality and transfer learning architecture. Health science over the years focused on artificial intelligence tools for care delivery, chronic care management, prevention/wellness, clinical supports, and diagnosis. The outcome of their research leads to cardiac arrest diagnosis through Heart Signal Computer-Aided Diagnostic tool (CADX) and other multifunctional deep learning techniques that offer care, diagnosis \& treatment. Health informatics provides monitored outcomes of human body organs through medical images that classify interstitial lung disease, detects image nodules for reconstruction \& tumor segmentation. The emergent medical research applications gave rise to clinical-pathological human-level performing tools for handling Radiological, Ophthalmological, and Dental diagnosis. This research will evaluate methodologies, Deep learning architectures, approaches, bio-informatics, specified function requirements, monitoring tools, ANN (artificial neural network), data labeling \& annotation algorithms that control data validation, modeling, and diagnosis of different diseases using smart monitoring health informatics applications.

Recent analyses of self-supervised learning (SSL) find the following data-centric properties to be critical for learning good representations: invariance to task-irrelevant semantics, separability of classes in some latent space, and recoverability of labels from augmented samples. However, given their discrete, non-Euclidean nature, graph datasets and graph SSL methods are unlikely to satisfy these properties. This raises the question: how do graph SSL methods, such as contrastive learning (CL), work well? To systematically probe this question, we perform a generalization analysis for CL when using generic graph augmentations (GGAs), with a focus on data-centric properties. Our analysis yields formal insights into the limitations of GGAs and the necessity of task-relevant augmentations. As we empirically show, GGAs do not induce task-relevant invariances on common benchmark datasets, leading to only marginal gains over naive, untrained baselines. Our theory motivates a synthetic data generation process that enables control over task-relevant information and boasts pre-defined optimal augmentations. This flexible benchmark helps us identify yet unrecognized limitations in advanced augmentation techniques (e.g., automated methods). Overall, our work rigorously contextualizes, both empirically and theoretically, the effects of data-centric properties on augmentation strategies and learning paradigms for graph SSL.

In recent years, face detection algorithms based on deep learning have made great progress. These algorithms can be generally divided into two categories, i.e. two-stage detector like Faster R-CNN and one-stage detector like YOLO. Because of the better balance between accuracy and speed, one-stage detectors have been widely used in many applications. In this paper, we propose a real-time face detector based on the one-stage detector YOLOv5, named YOLO-FaceV2. We design a Receptive Field Enhancement module called RFE to enhance receptive field of small face, and use NWD Loss to make up for the sensitivity of IoU to the location deviation of tiny objects. For face occlusion, we present an attention module named SEAM and introduce Repulsion Loss to solve it. Moreover, we use a weight function Slide to solve the imbalance between easy and hard samples and use the information of the effective receptive field to design the anchor. The experimental results on WiderFace dataset show that our face detector outperforms YOLO and its variants can be find in all easy, medium and hard subsets. Source code in https://github.com/Krasjet-Yu/YOLO-FaceV2

Mapping near-field pollutant concentration is essential to track accidental toxic plume dispersion in urban areas. By solving a large part of the turbulence spectrum, large-eddy simulations (LES) have the potential to accurately represent pollutant concentration spatial variability. Finding a way to synthesize this large amount of information to improve the accuracy of lower-fidelity operational models (e.g. providing better turbulence closure terms) is particularly appealing. This is a challenge in multi-query contexts, where LES become prohibitively costly to deploy to understand how plume flow and tracer dispersion change with various atmospheric and source parameters. To overcome this issue, we propose a non-intrusive reduced-order model combining proper orthogonal decomposition (POD) and Gaussian process regression (GPR) to predict LES field statistics of interest associated with tracer concentrations. GPR hyperpararameters are optimized component-by-component through a maximum a posteriori (MAP) procedure informed by POD. We provide a detailed analysis of the reducedorder model performance on a two-dimensional case study corresponding to a turbulent atmospheric boundary-layer flow over a surface-mounted obstacle. We show that near-source concentration heterogeneities upstream of the obstacle require a large number of POD modes to be well captured. We also show that the component-by-component optimization allows to capture the range of spatial scales in the POD modes, especially the shorter concentration patterns in the high-order modes. The reduced-order model predictions remain acceptable if the learning database is made of at least fifty to hundred LES snapshot providing a first estimation of the required budget to move towards more realistic atmospheric dispersion applications.

While it is well known that water availability controls vegetation growth and soil microbial activity, how aridity affects ecosystem carbon patterns is not completely understood. Towards a more quantitative assessment of terrestrial carbon stocks, here we uncover a remarkable self-similar behavior of the global carbon stock. Using international survey and remote sensing data, we find that the key statistics (e.g., mean, quantiles, and standard deviation) of carbon stock tend to scale with the hydrological regime (i.e., aridity) via a universal exponent. As a result, when normalized by its averages in the corresponding hydrological regime, the carbon stock distributions collapse onto a single curve. Such a scaling reflects the strong coupling between hydrological cycle and biogeochemical process and enables robust predictions of carbon stocks as a function of aridity only.

Recently, numerous efficient Transformers have been proposed to reduce the quadratic computational complexity of standard Transformers caused by the Softmax attention. However, most of them simply swap Softmax with an efficient attention mechanism without considering the customized architectures specially for the efficient attention. In this paper, we argue that the handcrafted vanilla Transformer architectures for Softmax attention may not be suitable for efficient Transformers. To address this issue, we propose a new framework to find optimal architectures for efficient Transformers with the neural architecture search (NAS) technique. The proposed method is validated on popular machine translation and image classification tasks. We observe that the optimal architecture of the efficient Transformer has the reduced computation compared with that of the standard Transformer, but the general accuracy is less comparable. It indicates that the Softmax attention and efficient attention have their own distinctions but neither of them can simultaneously balance the accuracy and efficiency well. This motivates us to mix the two types of attention to reduce the performance imbalance. Besides the search spaces that commonly used in existing NAS Transformer approaches, we propose a new search space that allows the NAS algorithm to automatically search the attention variants along with architectures. Extensive experiments on WMT' 14 En-De and CIFAR-10 demonstrate that our searched architecture maintains comparable accuracy to the standard Transformer with notably improved computational efficiency.

Land cover classification is a multi-class segmentation task to classify each pixel into a certain natural or man-made category of the earth surface, such as water, soil, natural vegetation, crops, and human infrastructure. Limited by hardware computational resources and memory capacity, most existing studies preprocessed original remote sensing images by down sampling or cropping them into small patches less than 512*512 pixels before sending them to a deep neural network. However, down sampling images incurs spatial detail loss, renders small segments hard to discriminate, and reverses the spatial resolution progress obtained by decades of years of efforts. Cropping images into small patches causes a loss of long-range context information, and restoring the predicted results to their original size brings extra latency. In response to the above weaknesses, we present an efficient lightweight semantic segmentation network termed MKANet. Aimed at the characteristics of top view high-resolution remote sensing imagery, MKANet utilizes sharing kernels to simultaneously and equally handle ground segments of inconsistent scales, and also employs parallel and shallow architecture to boost inference speed and friendly support image patches more than 10X larger. To enhance boundary and small segments discrimination, we also propose a method that captures category impurity areas, exploits boundary information and exerts an extra penalty on boundaries and small segment misjudgment. Both visual interpretations and quantitative metrics of extensive experiments demonstrate that MKANet acquires state-of-the-art accuracy on two land-cover classification datasets and infers 2X faster than other competitive lightweight networks. All these merits highlight the potential of MKANet in practical applications.

This chapter describes the scientific motivations that led to the development of the SMILE (Solar wind Magnetosphere Ionosphere Link Explorer) mission. The solar wind coupling with the terrestrial magnetosphere is a key link in Sun-Earth interactions. In-situ missions can provide detailed observations of plasma and magnetic field conditions in the solar wind and the magnetosphere, but leave us still unable to fully quantify the global effects of the drivers of Sun-Earth connections, and to monitor their evolution. This information is essential to develop a comprehensive understanding of how the Sun controls the Earth's plasma environment and space weather. SMILE offers a new approach to global monitoring of geospace by imaging the magnetosheath and cusps in X-rays emitted when high charge-state solar wind ions exchange charges with exospheric neutrals. SMILE combines this with simultaneous UV imaging of the northern aurora and in-situ plasma and magnetic field measurements in the magnetosheath and solar wind from a highly elliptical northern polar orbit. In this chapter the science that SMILE will explore and the scientific preparations that will ensure the optimal exploitation of SMILE measurements are presented.

Two-dimensional (2D) transition metal dichalcogenides (TMDC) and their moir\'e interfaces have been demonstrated for correlated electron states, including Mott insulators and electron/hole crystals commensurate with moir\'e superlattices. Here we present spectroscopic evidences for ordered bosons - interlayer exciton crystals in a WSe2/MoSe2/WSe2 trilayer, where the enhanced Coulomb interactions over those in heterobilayers have been predicted to result in exciton ordering. While the dipolar interlayer excitons in the heterobilayer may be ordered by the periodic moir\'e traps, their mutual repulsion results in de-trapping at exciton density n_ex larger than 10^11 cm^-2 to form mobile exciton gases and further to electron-hole plasmas, both accompanied by broadening in photoluminescence (PL) peaks and large increases in mobility. In contrast, ordered interlayer excitons in the trilayer are characterized by negligible mobility and by sharper PL peaks persisting to n_ex approximately 10^12 cm^-2. We present evidences for the predicted quadrupolar exciton crystal and its transitions to dipolar excitons either with increasing n_ex or by an applied electric field. These ordered interlayer excitons may serve as models for the exploration of quantum phase transitions and quantum coherent phenomena.

This work presents SeasoNet, a new large-scale multi-label land cover and land use scene understanding dataset. It includes $1\,759\,830$ images from Sentinel-2 tiles, with 12 spectral bands and patch sizes of up to $ 120 \ \mathrm{px} \times 120 \ \mathrm{px}$. Each image is annotated with large scale pixel level labels from the German land cover model LBM-DE2018 with land cover classes based on the CORINE Land Cover database (CLC) 2018 and a five times smaller minimum mapping unit (MMU) than the original CLC maps. We provide pixel synchronous examples from all four seasons, plus an additional snowy set. These properties make SeasoNet the currently most versatile and biggest remote sensing scene understanding dataset with possible applications ranging from scene classification over land cover mapping to content-based cross season image retrieval and self-supervised feature learning. We provide baseline results by evaluating state-of-the-art deep networks on the new dataset in scene classification and semantic segmentation scenarios.

The Australian continent contributes substantially to the year-to-year variability of the global terrestrial carbon dioxide (CO2) sink. However, the scarcity of in-situ observations in remote areas prevents deciphering the processes that force the CO2 flux variability. Here, examining atmospheric CO2 measurements from satellites in the period 2009-2018, we find recurrent end-of-dry-season CO2 pulses over the Australian continent. These pulses largely control the year-to-year variability of Australia's CO2 balance, due to 2-3 times higher seasonal variations compared to previous top-down inversions and bottom-up estimates. The CO2 pulses occur shortly after the onset of rainfall and are driven by enhanced soil respiration preceding photosynthetic uptake in Australia's semi-arid regions. The suggested continental-scale relevance of soil rewetting processes has large implications for our understanding and modelling of global climate-carbon cycle feedbacks.

The explosion in novel NLP word embedding and deep learning techniques has induced significant endeavors into potential applications. One of these directions is in the financial sector. Although there is a lot of work done in state-of-the-art models like GPT and BERT, there are relatively few works on how well these methods perform through fine-tuning after being pre-trained, as well as info on how sensitive their parameters are. We investigate the performance and sensitivity of transferred neural architectures from pre-trained GPT-2 and BERT models. We test the fine-tuning performance based on freezing transformer layers, batch size, and learning rate. We find the parameters of BERT are hypersensitive to stochasticity in fine-tuning and that GPT-2 is more stable in such practice. It is also clear that the earlier layers of GPT-2 and BERT contain essential word pattern information that should be maintained.

Relative sea level records climatic change as well as vertical land movement. In Barbados, uplift variation is necessary to interpret one of the most complete coral reef records. Here we show that an abrupt mass unloading of 30 km3 caused an uplift variation of ~0.45 mm/yr using a modelling approach. Simulations have been conducted for different volumes and elastic thicknesses. Isostatic adjustment in relation with an abrupt mass unloading explains the observed uplift rate increased from 0.34 mm/yr to 0.8 mm/yr that occurred 11.2 kyr ago. The reconstructed sea-level curve highlights a sea-level jump of 4.8 m, with a delay of 150 yr from the termination of Younger Dryas cold event and 300 yr before the abrupt mass unloading. This sea-level jump corresponds to meltwater pulse MWP-1B and is not an artefact. A stagnation of 500 yr occurred from 12 to 11.5 kyr BP. Relative sea level records are useful to detect past landslides and erosion. Accurate analysis and reconstruction of sea-level permits to determine sea-level abrupt rise caused by climate warming during the last thousand years.

A typical trajectory planner of autonomous driving commonly relies on predicting the future behavior of surrounding obstacles. Recently, deep learning technology has been widely adopted to design prediction models due to their impressive performance. However, such models may fail in the "long-tail" driving cases where the training data is sparse or unavailable, leading to planner failures. To this end, this work proposes a trajectory planner to consider the prediction model uncertainty arising from insufficient data for safer performance. Firstly, an ensemble network structure estimates the prediction model's uncertainty due to insufficient training data. Then a trajectory planner is designed to consider the worst-case arising from prediction uncertainty. The results show that the proposed method can improve the safety of trajectory planning under the prediction uncertainty caused by insufficient data. At the same time, with sufficient data, the framework will not lead to overly conservative results. This technology helps to improve the safety and reliability of autonomous vehicles under the long-tail data distribution of the real world.

In deep learning research, self-supervised learning (SSL) has received great attention triggering interest within both the computer vision and remote sensing communities. While there has been a big success in computer vision, most of the potential of SSL in the domain of earth observation remains locked. In this paper, we provide an introduction to, and a review of the concepts and latest developments in SSL for computer vision in the context of remote sensing. Further, we provide a preliminary benchmark of modern SSL algorithms on popular remote sensing datasets, verifying the potential of SSL in remote sensing and providing an extended study on data augmentations. Finally, we identify a list of promising directions of future research in SSL for earth observation (SSL4EO) to pave the way for fruitful interaction of both domains.

The increasing number of Anthropogenic Space Objects (ASOs) in Low Earth Orbit (LEO) poses a threat to the safety and sustainability of the space environment. Multiple companies are planning to launch large constellations of hundreds or thousands of satellites in the near future, increasing congestion in LEO and the risk of collisions and debris generation. This paper employs a new multi-shell multi-species evolutionary source-sink model, called MOCAT-3, to estimate LEO orbital capacity. In particular, a new definition of orbital capacity based on the stable equilibrium points of the system is provided. Moreover, an optimization approach is used to compute the maximum orbital capacity of the low region of LEO (200-900 km of altitude), considering the equilibrium solutions and the failure rate of satellites as a constraint. Hence, an estimate for the maximum number of satellites that it is possible to fit in LEO, considering the stability of the space environment, is obtained. As a result, considering 7% of failure rate, the maximum orbital capacity of LEO is estimated to be about 12.6 million satellites. Compatibility of future traffic launch, especially in terms of satellite constellations, is also analyzed and a strategy to accommodate for future traffic needs is proposed.

Unets have become the standard method for semantic segmentation of medical images, along with fully convolutional networks (FCN). Unet++ was introduced as a variant of Unet, in order to solve some of the problems facing Unet and FCNs. Unet++ provided networks with an ensemble of variable depth Unets, hence eliminating the need for professionals estimating the best suitable depth for a task. While Unet and all its variants, including Unet++ aimed at providing networks that were able to train well without requiring large quantities of annotated data, none of them attempted to eliminate the need for pixel-wise annotated data altogether. Obtaining such data for each disease to be diagnosed comes at a high cost. Hence such data is scarce. In this paper we use contrastive learning to train Unet++ for semantic segmentation of medical images using medical images from various sources including magnetic resonance imaging (MRI) and computed tomography (CT), without the need for pixel-wise annotations. Here we describe the architecture of the proposed model and the training method used. This is still a work in progress and so we abstain from including results in this paper. The results and the trained model would be made available upon publication or in subsequent versions of this paper on arxiv.

While annotating decent amounts of data to satisfy sophisticated learning models can be cost-prohibitive for many real-world applications. Active learning (AL) and semi-supervised learning (SSL) are two effective, but often isolated, means to alleviate the data-hungry problem. Some recent studies explored the potential of combining AL and SSL to better probe the unlabeled data. However, almost all these contemporary SSL-AL works use a simple combination strategy, ignoring SSL and AL's inherent relation. Further, other methods suffer from high computational costs when dealing with large-scale, high-dimensional datasets. Motivated by the industry practice of labeling data, we propose an innovative Inconsistency-based virtual aDvErsarial Active Learning (IDEAL) algorithm to further investigate SSL-AL's potential superiority and achieve mutual enhancement of AL and SSL, i.e., SSL propagates label information to unlabeled samples and provides smoothed embeddings for AL, while AL excludes samples with inconsistent predictions and considerable uncertainty for SSL. We estimate unlabeled samples' inconsistency by augmentation strategies of different granularities, including fine-grained continuous perturbation exploration and coarse-grained data transformations. Extensive experiments, in both text and image domains, validate the effectiveness of the proposed algorithm, comparing it against state-of-the-art baselines. Two real-world case studies visualize the practical industrial value of applying and deploying the proposed data sampling algorithm.

The Bay of Bengal (BoB) is a high recipient of freshwater flux from rivers and precipitation, making the region strongly stratified. The strong stratification results in a thick barrier layer formation, which inhibits vertical mixing making this region a low-productive zone. In the present study, we attempt to model the pH of the BoB region and understand the role of different governing factors such as sea-surface temperature (SST), sea-surface salinity (SSS), dissolved inorganic carbon (DIC), and total alkalinity (TALK) on the seasonality of sea-surface pH. We run a set of sensitivity experiments to understand the role of each of the governing factors. The results show that the SST, SSS, and DIC are the principal drivers affecting the sea-surface pH, while TALK plays a buffering role. The SST and DIC are consistently found to be opposite to each other. The pre-monsoon season (MAM) has shown to have an almost equal contribution from all the drivers. In the pre-monsoon season, the SST and DIC are balanced by TALK and SSS. The role of SSS is significantly dominant in the second half of the year. Both SST and SSS counter the role of DIC in the southwest monsoon season. The strong stratification plays an essential role in modulating the pH of the BoB region. The thickness of the barrier layer formed in the sub-surface layers positively affects the sea-surface pH. The northern BoB is found to be more alkaline than the southern BoB. Our study highlights the complexity of ocean acidification in the BoB region compared to the other part of the world ocean.

Sentence scoring aims at measuring the likelihood score of a sentence and is widely used in many natural language processing scenarios, like reranking, which is to select the best sentence from multiple candidates. Previous works on sentence scoring mainly adopted either causal language modeling (CLM) like GPT or masked language modeling (MLM) like BERT, which have some limitations: 1) CLM only utilizes unidirectional information for the probability estimation of a sentence without considering bidirectional context, which affects the scoring quality; 2) MLM can only estimate the probability of partial tokens at a time and thus requires multiple forward passes to estimate the probability of the whole sentence, which incurs large computation and time cost. In this paper, we propose \textit{Transcormer} -- a Transformer model with a novel \textit{sliding language modeling} (SLM) for sentence scoring. Specifically, our SLM adopts a triple-stream self-attention mechanism to estimate the probability of all tokens in a sentence with bidirectional context and only requires a single forward pass. SLM can avoid the limitations of CLM (only unidirectional context) and MLM (multiple forward passes) and inherit their advantages, and thus achieve high effectiveness and efficiency in scoring. Experimental results on multiple tasks demonstrate that our method achieves better performance than other language modelings.

Self-supervised learning (SSL) has been extensively explored in recent years. Particularly, generative SSL has seen emerging success in natural language processing and other AI fields, such as the wide adoption of BERT and GPT. Despite this, contrastive learning-which heavily relies on structural data augmentation and complicated training strategies-has been the dominant approach in graph SSL, while the progress of generative SSL on graphs, especially graph autoencoders (GAEs), has thus far not reached the potential as promised in other fields. In this paper, we identify and examine the issues that negatively impact the development of GAEs, including their reconstruction objective, training robustness, and error metric. We present a masked graph autoencoder GraphMAE that mitigates these issues for generative self-supervised graph pretraining. Instead of reconstructing graph structures, we propose to focus on feature reconstruction with both a masking strategy and scaled cosine error that benefit the robust training of GraphMAE. We conduct extensive experiments on 21 public datasets for three different graph learning tasks. The results manifest that GraphMAE-a simple graph autoencoder with careful designs-can consistently generate outperformance over both contrastive and generative state-of-the-art baselines. This study provides an understanding of graph autoencoders and demonstrates the potential of generative self-supervised pre-training on graphs.

Existing pre-trained transformer analysis works usually focus only on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. In our work, we utilize the oLMpics benchmark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. Additionally, we adapt the oLMpics zero-shot setup for autoregressive models and evaluate GPT networks of different sizes. Our findings show that none of these models can resolve compositional questions in a zero-shot fashion, suggesting that this skill is not learnable using existing pre-training objectives. Furthermore, we find that global model decisions such as architecture, directionality, size of the dataset, and pre-training objective are not predictive of a model's linguistic capabilities.

Developments in reinforcement learning (RL) have allowed algorithms to achieve impressive performance in highly complex, but largely static problems. In contrast, biological learning seems to value efficiency of adaptation to a constantly-changing world. Here we build on a recently-proposed neuronal learning rule that assumes each neuron can optimize its energy balance by predicting its own future activity. That assumption leads to a neuronal learning rule that uses presynaptic input to modulate prediction error. We argue that an analogous RL rule would use action probability to modulate reward prediction error. This modulation makes the agent more sensitive to negative experiences, and more careful in forming preferences. We embed the proposed rule in both tabular and deep-Q-network RL algorithms, and find that it outperforms conventional algorithms in simple, but highly-dynamic tasks. We suggest that the new rule encapsulates a core principle of biological intelligence; an important component for allowing algorithms to adapt to change in a human-like way.

In multi-agent systems, noise reduction techniques are important for improving the overall system reliability as agents are required to rely on limited environmental information to develop cooperative and coordinated behaviors with the surrounding agents. However, previous studies have often applied centralized noise reduction methods to build robust and versatile coordination in noisy multi-agent environments, while distributed and decentralized autonomous agents are more plausible for real-world application. In this paper, we introduce a \emph{distributed attentional actor architecture model for a multi-agent system} (DA3-X), using which we demonstrate that agents with DA3-X can selectively learn the noisy environment and behave cooperatively. We experimentally evaluate the effectiveness of DA3-X by comparing learning methods with and without DA3-X and show that agents with DA3-X can achieve better performance than baseline agents. Furthermore, we visualize heatmaps of \emph{attentional weights} from the DA3-X to analyze how the decision-making process and coordinated behavior are influenced by noise.

Chlorophyll-a (Chl) retrieval from ocean colour remote sensing is problematic for relatively turbid coastal waters due to the impact of non-algal materials on atmospheric correction and standard Chl algorithm performance. Artificial neural networks (NNs) provide an alternative approach for retrieval of Chl from space and results in northwest European shelf seas over the 2002-2020 period are shown. The NNs operate on 15 MODIS-Aqua visible and infrared bands and are tested using bottom of atmosphere (BOA), top of atmosphere (TOA) and Rayleigh corrected TOA reflectances (RC). In each case, a NN architecture consisting of 3 layers of 15 neurons improved performances and data availability compared to current state-of-the-art algorithms used in the region. The NN operating on TOA reflectance outperformed BOA and RC versions. By operating on TOA reflectance data, the NN approach overcomes the common but difficult problem of atmospheric correction in coastal waters. Moreover, the NN provides data for regions which other algorithms often mask out for turbid water or low zenith angle flags. A distinguishing feature of the NN approach is generation of associated product uncertainties based on multiple resampling of the training data set to produce a distribution of values for each pixel, and an example is shown for a coastal time series in the North Sea. The final output of the NN approach consists of a best-estimate image based on medians for each pixel and a second image representing uncertainty based on standard deviation for each pixel, providing pixel-specific estimates of uncertainty in the final product.

The Anthologies of the second-century astrologer Vettius Valens (120-c.175 CE) is the most extensive surviving practical astrological text from the period. Despite this, the theoretical underpinnings of the Anthologies have been understudied; in general, the work has been overshadowed by Ptolemy's contemporaneous Tetrabiblos. While the Tetrabiblos explicitly aims to present a systematic account of astrology, Valens' work is often characterised as a miscellaneous collection, of interest to historians only for the evidence it preserves about the practical methods used in casting horoscopes. In this article, we argue that the Anthologies is also an invaluable resource for engagement with the conceptual basis of astrology. As a case study, we take a section of Anthologies Book 1 which lists the possible astrological effects of planets, both alone and in 'combinations' of two and three. We demonstrate that analysing Valens' descriptions quantitatively with textual analysis reveals a consistent internal logic of planetary combination. By classifying descriptive terms as positive or negative, we show that the resulting 'sentiment' of planetary combinations is well-correlated with their component parts. Furthermore, we find that the sentiment of three-planet combinations is more strongly correlated with the average sentiment of their three possible component pairs than with the average sentiment of individual planets, suggesting an iterative combinatorial logic. Recognition of this feature of astrological practice has been neglected compared to the mathematical methods for calculating horoscopes. We argue that this analysis not only provides evidence that the astrological lore detailed in Valens is more consistent than is often assumed, but is also indicative of a wider methodological technique in practical astrology: combinatorial reasoning from existing astrological lore.

Dual embodied-symbolic concept representations are the foundation for deep learning and symbolic AI integration. We discuss the use of dual embodied-symbolic concept representations for molecular graph representation learning, specifically with exemplar-based contrastive self-supervised learning (SSL). The embodied representations are learned from molecular graphs, and the symbolic representations are learned from the corresponding Chemical knowledge graph (KG). We use the Chemical KG to enhance molecular graphs with symbolic (semantic) knowledge and generate their augmented molecular graphs. We treat a molecular graph and its semantically augmented molecular graph as exemplars of the same semantic class, and use the pairs as positive pairs in exemplar-based contrastive SSL.

Artificial Intelligence (AI) is a fast-growing research and development (R&D) discipline which is attracting increasing attention because of its promises to bring vast benefits for consumers and businesses, with considerable benefits promised in productivity growth and innovation. To date it has reported significant accomplishments in many areas that have been deemed as challenging for machines, ranging from computer vision, natural language processing, audio analysis to smart sensing and many others. The technical trend in realizing the successes has been towards increasing complex and large size AI models so as to solve more complex problems at superior performance and robustness. This rapid progress, however, has taken place at the expense of substantial environmental costs and resources. Besides, debates on the societal impacts of AI, such as fairness, safety and privacy, have continued to grow in intensity. These issues have presented major concerns pertaining to the sustainable development of AI. In this work, we review major trends in machine learning approaches that can address the sustainability problem of AI. Specifically, we examine emerging AI methodologies and algorithms for addressing the sustainability issue of AI in two major aspects, i.e., environmental sustainability and social sustainability of AI. We will also highlight the major limitations of existing studies and propose potential research challenges and directions for the development of next generation of sustainable AI techniques. We believe that this technical review can help to promote a sustainable development of AI R&D activities for the research community.

Clouds play a key role in regulating climate change but are difficult to simulate within Earth system models (ESMs). Improving the representation of clouds is one of the key tasks towards more robust climate change projections. This study introduces a new machine-learning based framework relying on satellite observations to improve understanding of the representation of clouds and their relevant processes in climate models. The proposed method is capable of assigning distributions of established cloud types to coarse data. It facilitates a more objective evaluation of clouds in ESMs and improves the consistency of cloud process analysis. The method is built on satellite data from the MODIS instrument labelled by deep neural networks with cloud types defined by the World Meteorological Organization (WMO), using cloud type labels from CloudSat as ground truth. The method is applicable to datasets with information about physical cloud variables comparable to MODIS satellite data and at sufficiently high temporal resolution. We apply the method to alternative satellite data from the Cloud\_cci project (ESA Climate Change Initiative), coarse-grained to typical resolutions of climate models. The resulting cloud type distributions are physically consistent and the horizontal resolutions typical of ESMs are sufficient to apply our method. We recommend outputting crucial variables required by our method for future ESM data evaluation. This will enable the use of labelled satellite data for a more systematic evaluation of clouds in climate models.

Many wind wave spectrum models provide excellent wave height prediction given the input of wind speed and wave age. Their quantification of the surface roughness, on the other hand, varies considerably. The ocean surface roughness is generally represented by the mean square slope, its direct measurement in open ocean remains a challenging task. Microwave ocean remote sensing from space delivers ocean surface roughness information. Satellite platforms offer global coverage in a broad range of environmental conditions. This paper presents lowpass mean square slope (LPMSS) data obtained by spaceborne microwave altimeters and reflectometers operating at L, Ku, and Ka bands (about 1.6, 14, and 36 GHz). The LPMSS data represent the spectrally integrated ocean surface roughness with 11, 95, and 250 rad/m upper cutoff wave numbers, the maximum wind speeds are 80, 29, and 25 m/s, respectively. A better understanding of the ocean surface roughness is important to the goal of improving wind wave spectrum modeling. The analysis presented in this paper shows that over two orders of magnitude of the wave number range (0.3 to 30 rad/m), the spectral components follow a power function relating the dimensionless spectrum and the ratio between wind friction velocity and wave phase speed. The power function exponent is 0.38, which is considerable smaller than unity as expected from the classical equilibrium spectrum function. It may suggest that wave breaking is not only an energy sink but also a source of roughness generation covering a wideband of wavelengths about 20 m and shorter.

The development of analytical software for big Earth observation data faces several challenges. Designers need to balance between conflicting factors. Solutions that are efficient for specific hardware architectures can not be used in other environments. Packages that work on generic hardware and open standards will not have the same performance as dedicated solutions. Software that assumes that its users are computer programmers are flexible but may be difficult to learn for a wide audience. This paper describes sits, an open-source R package for satellite image time series analysis using machine learning. To allow experts to use satellite imagery to the fullest extent, sits adopts a time-first, space-later approach. It supports the complete cycle of data analysis for land classification. Its API provides a simple but powerful set of functions. The software works in different cloud computing environments. Satellite image time series are input to machine learning classifiers, and the results are post-processed using spatial smoothing. Since machine learning methods need accurate training data, sits includes methods for quality assessment of training samples. The software also provides methods for validation and accuracy measurement. The package thus comprises a production environment for big EO data analysis. We show that this approach produces high accuracy for land use and land cover maps through a case study in the Cerrado biome, one of the world's fast moving agricultural frontiers for the year 2018.

Since 2017, the Transformer-based models play critical roles in various downstream Natural Language Processing tasks. However, a common limitation of the attention mechanism utilized in Transformer Encoder is that it cannot automatically capture the information of word order, so explicit position embeddings are generally required to be fed into the target model. In contrast, Transformer Decoder with the causal attention masks is naturally sensitive to the word order. In this work, we focus on improving the position encoding ability of BERT with the causal attention masks. Furthermore, we propose a new pre-trained language model DecBERT and evaluate it on the GLUE benchmark. Experimental results show that (1) the causal attention mask is effective for BERT on the language understanding tasks; (2) our DecBERT model without position embeddings achieve comparable performance on the GLUE benchmark; and (3) our modification accelerates the pre-training process and DecBERT w/ PE achieves better overall performance than the baseline systems when pre-training with the same amount of computational resources.

In this paper, for a henselian valued field $(K, v)$ of arbitrary rank and an extension $w$ of $v$ to $K(X),$ we use abstract key polynomials for $w$ to give a connection between complete sets, saturated distinguished chains and Okutsu frames. Further, for a valued field $(K, v),$ we also obtain a close connection between complete set of ABKPs for $w$ and Maclane-Vaqui\'e chains of $w.$

Self-supervised learning (SSL) has attracted much interest in remote sensing and earth observation due to its ability to learn task-agnostic representations without human annotation. While most of the existing SSL works in remote sensing utilize ConvNet backbones and focus on a single modality, we explore the potential of vision transformers (ViTs) for joint SAR-optical representation learning. Based on DINO, a state-of-the-art SSL algorithm that distills knowledge from two augmented views of an input image, we combine SAR and optical imagery by concatenating all channels to a unified input. Subsequently, we randomly mask out channels of one modality as a data augmentation strategy. While training, the model gets fed optical-only, SAR-only, and SAR-optical image pairs learning both inner- and intra-modality representations. Experimental results employing the BigEarthNet-MM dataset demonstrate the benefits of both, the ViT backbones and the proposed multimodal SSL algorithm DINO-MM.

Machine learning, satellites or local sensors are key factors for a sustainable and resource-saving optimisation of agriculture and proved its values for the management of agricultural land. Up to now, the main focus was on the enlargement of data which were evaluated by means of supervised learning methods. Nevertheless, the need for labels is also a limiting and time-consuming factor, while in contrast, ongoing technological development is already providing an ever-increasing amount of unlabeled data. Self-supervised learning (SSL) could overcome this limitation and incorporate existing unlabeled data. Therefore, a crop type data set was utilized to conduct experiments with SSL and compare it to supervised methods. A unique feature of our data set from 2016 to 2018 was a divergent climatological condition in 2018 that reduced yields and affected the spectral fingerprint of the plants. Our experiments focused on predicting 2018 using SLL without or a few labels to clarify whether new labels should be collected for an unknown year. Despite these challenging conditions, the results showed that SSL contributed to higher accuracies. We believe that the results will encourage further improvements in the field of precision farming, why the SSL framework and data will be published (Marszalek, 2021).

Reducing soil salinization of croplands with optimized irrigation and water management is essential to achieve land degradation neutralization. The effectiveness and sustainability of various irrigation and water management measures to reduce basin-scale salinization remain uncertain. Here we use remote sensing to estimate soil salinity of arid croplands from 1984 to 2018. We then use Bayesian network analysis to compare the spatial-temporal response of salinity to water management, including various irrigation and drainage methods, in ten large arid river basins: Nile, Tigris-Euphrates, Indus, Tarim, Amu, Ili, Syr, Junggar, Colorado, and San Joaquin. Managers in basins at more advanced phases of development implemented drip and groundwater irrigation, which effectively controlled salinity by lowering groundwater levels. For the remaining basins where conventional flood irrigation is used, economic development and policies are crucial to establishing a virtuous circle of improving irrigation systems, reducing salinity, and increasing agricultural incomes necessary to achieve LDN.

A model of the extent of wave driven overwash into fields of sea ice floes is proposed. The extent model builds on previous work modelling wave overwash of a single floe by regular waves by including irregular incoming waves and random floe fields. The model is validated against a laboratory experiment. It is then used to study the extent of wave overwash into marginal ice zones consisting of pancake and fragmented floe fields. The effects of wave conditions and floe geometry on predicted extents are investigated. Finally, the model is used to predict the wave overwash extent for the conditions observed during a winter (July) 2017 Antarctic voyage in which the sea surface was monitored by a stereo-camera system.

Axle count in trucks is important to the classification of vehicles and to the operation of road systems. It is used in the determination of service fees and in the impact on the pavement. Although axle count can be achieved with traditional methods, such as manual labor, it is increasingly possible to count axles using deep learning and computer vision methods. This paper aims to compare three deep-learning object detection algorithms, YOLO, Faster R-CNN, and SSD, for the detection of truck axles. A dataset was built to provide training and testing examples for the neural networks. The training was done on different base models, to increase training time efficiency and to compare results. We evaluated results based on five metrics: precision, recall, mAP, F1-score, and FPS count. Results indicate that YOLO and SSD have similar accuracy and performance, with more than 96\% mAP for both models. Datasets and codes are publicly available for download.

Safe exploration is a common problem in reinforcement learning (RL) that aims to prevent agents from making disastrous decisions while exploring their environment. A family of approaches to this problem assume domain knowledge in the form of a (partial) model of this environment to decide upon the safety of an action. A so-called shield forces the RL agent to select only safe actions. However, for adoption in various applications, one must look beyond enforcing safety and also ensure the applicability of RL with good performance. We extend the applicability of shields via tight integration with state-of-the-art deep RL, and provide an extensive, empirical study in challenging, sparse-reward environments under partial observability. We show that a carefully integrated shield ensures safety and can improve the convergence rate and final performance of RL agents. We furthermore show that a shield can be used to bootstrap state-of-the-art RL agents: they remain safe after initial learning in a shielded setting, allowing us to disable a potentially too conservative shield eventually.

The Internet of Things (IoT) boom has revolutionized almost every corner of people's daily lives: healthcare, home, transportation, manufacturing, supply chain, and so on. With the recent development of sensor and communication technologies, IoT devices including smart wearables, cameras, smartwatches, and autonomous vehicles can accurately measure and perceive their surrounding environment. Continuous sensing generates massive amounts of data and presents challenges for machine learning. Deep learning models (e.g., convolution neural networks and recurrent neural networks) have been extensively employed in solving IoT tasks by learning patterns from multi-modal sensory data. Graph Neural Networks (GNNs), an emerging and fast-growing family of neural network models, can capture complex interactions within sensor topology and have been demonstrated to achieve state-of-the-art results in numerous IoT learning tasks. In this survey, we present a comprehensive review of recent advances in the application of GNNs to the IoT field, including a deep dive analysis of GNN design in various IoT sensing environments, an overarching list of public data and source code from the collected publications, and future research directions. To keep track of newly published works, we collect representative papers and their open-source implementations and create a Github repository at https://github.com/GuiminDong/GNN4IoT.

Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, deploying these models can be prohibitively costly, as the standard self-attention mechanism of the Transformer suffers from quadratic computational cost in the input sequence length. To confront this, we propose FCA, a fine- and coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention. Specifically, FCA conducts an attention-based scoring strategy to determine the informativeness of tokens at each layer. Then, the informative tokens serve as the fine-granularity computing units in self-attention and the uninformative tokens are replaced with one or several clusters as the coarse-granularity computing units in self-attention. Experiments on GLUE and RACE datasets show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with <1% loss in accuracy. We show that FCA offers a significantly better trade-off between accuracy and FLOPs compared to prior methods.

High spatial resolution PM2.5 data covering a long time period are urgently needed to support population exposure assessment and refined air quality management. In this study, we provided complete-coverage PM2.5 predictions with a 1-km spatial resolution from 2000 to the present under the Tracking Air Pollution in China (TAP, http://tapdata.org.cn/) framework. To support high spatial resolution modelling, we collected PM2.5 measurements from both national and local monitoring stations. To correctly reflect the temporal variations in land cover characteristics that affected the local variations in PM2.5, we constructed continuous annual geoinformation datasets, including the road maps and ensemble gridded population maps, in China from 2000 to 2021. We also examined various model structures and predictor combinations to balance the computational cost and model performance. The final model fused 10-km TAP PM2.5 predictions from our previous work, 1-km satellite aerosol optical depth retrievals and land use parameters with a random forest model. Our annual model had an out-of-bag R2 ranging between 0.80 and 0.84, and our hindcast model had a by-year cross-validation R2 of 0.76. This open-access 1-km resolution PM2.5 data product with complete coverage successfully revealed the local-scale spatial variations in PM2.5 and could benefit environmental studies and policy-making.

Fiber-optic sensing technologies based on transmission offer an alternative to scattering-based Distributed Acoustic Sensing (DAS). Being able to interrogate fibers that are thousands of kilometers long, opens opportunities for seismological studies of remote regions, including ocean basins. However, by averaging deformation along the fiber, transmission systems only produce integrated and not distributed measurements. Here we develop a formalism to calculate sensitivity kernels with respect to (Earth) structure, using optical phase delay measurements. With this, we demonstrate that transmission-based sensing can effectively provide distributed measurements when the phase delay time series is dissected into different windows. The extent to which a potentially useful sensitivity coverage can be achieved, depends on the fiber geometry, and specifically on its local curvature. This work establishes a theoretical foundation for both tomographic inversions and experimental design, using transmission-based optical sensing.